Senior Backend Engineer, AI Evaluations
- Salary
- $175K - $250K
- Level
- Senior
- Work type
- Hybrid · San Francisco
Not stated in the posting: visa.
Open. First seen 2 October 2026.
About the role
About the Role
We’re seeking a strong Senior Backend Engineer to join our AI Evaluation team, focused on building the infrastructure and internal tooling that enable reliable, repeatable evaluation of AI systems in production. In this role, you’ll develop the scaffolding around our evaluations platform - API/data integrations, test configuration management, and reporting tools that make evaluations easy to run, extend, and operate at scale. You’ll also build and maintain evaluation frameworks for individual system components such as ASR, LLMs, TTS, knowledge bases, and guardrails, productionalizing these frameworks to support regular analysis, regression detection, and continuous monitoring.
In addition, you’ll create debugging tools that make it easy to inspect end-to-end calls, trace failures, and surface all relevant signals in one place - empowering internal teams to diagnose issues quickly and confidently. The ideal candidate is a pragmatic, infrastructure-minded engineer who enjoys turning ad hoc analysis into durable systems, cares deeply about developer experience, holds a high bar for software-quality and takes pride in building tooling that makes complex AI systems observable, testable, and easier to operate.
Responsibilities:
- Build and maintain infrastructure and tooling for the AI evaluations platform used by internal teams, including automated testing platform for AI voice agents, debugging and observability tools.
- Develop and productionalize evaluation frameworks for individual system components such as ASR, LLMs, TTS, knowledge bases, and guardrails.
- Partner with ML, engineering and QA teams to translate evaluation requirements into robust, maintainable infrastructure and tooling.
- Improve developer experience by making evaluation systems easy to extend, well-documented, and reliable in day-to-day use.
- Ensure evaluation tooling meets production standards for reliability, performance, and maintainability.
Qualifications:
- 5+ years of professional software engineering experience, with a strong focus on building backend systems, platforms, or developer tooling.
- Proven experience designing and maintaining production-grade infrastructure with code, including APIs, services, and data pipelines.
- Strong proficiency in at least one general-purpose programming language (e.g., Python, Typescript/Javascript, Java, or similar).
- Experience using test automation frameworks, evaluation pipelines, or CI/CD-integrated testing systems.
- Familiarity with observability and debugging tools (logging, metrics, tracing) and building internal tools that improve developer and QA workflows.
- Strong debugging skills and a methodical approach to diagnosing production and evaluation issues.
- Ability to collaborate effectively across engineering, QA, and operations teams, translating requirements into reliable, maintainable systems.
- Product-minded approach to infrastructure, with attention to usability, documentation, and long-term maintainability.
Preferred:
- Experience working with complex, multi-component systems (e.g., ASR, LLMs, TTS, or other ML-powered services)
- Experience working in healthcare or other regulated environments, including awareness of HIPAA and PHI handling.
- Familiarity with conversational AI or voice agents, including multi-turn dialogue, latency constraints, and error recovery.
- Familiarity with LLM observability or evaluation tools (e.g., Langfuse, prompt eval frameworks).
- Background in digital health, care coordination, or patient-facing systems.
Salary and Benefits
We offer competitive salary and benefits, including 401(k) matching, health, vision, and dental insurance, and very flexible paid time off.
The typical salary range for this role is $175,000 to $250,000 USD, depending on skills, qualifications, and relevant experience.
Ellipsis Health's standard text
Background Checks
As a health technology company, we reserve the right to run background checks on candidates to whom we extend offers, in compliance with applicable laws. We evaluate candidates holistically and comply with all “ban the box” regulations.
Assistance
If you have a disability or require accommodations during the application or recruitment process, please contact [email protected].
What this role asks for
Particular to this role
Test Automation
Rare in Ellipsis Health's other 12 open postings.
Standard for backend roles
CI/CD · Java · Python
Common in backend postings at any company.
In most of Ellipsis Health's ads
LLM
Named in most of its open postings, whatever the role.
90% of this description is written for the role. The rest is Ellipsis Health's standard text, folded away below.
Compared with Ellipsis Health's other open jobs.
The team: Engineering HIRES STEADILY
This role is one of 8 open in Engineering. 3 were posted in the last 30 days and the oldest in May: a team that hires all year rather than one being built.
WHAT THE TEAM WORKS WITH, ACROSS ITS POSTINGS
AWS (5 of 8), CI/CD (5 of 8), GCP (5 of 8), Kubernetes (5 of 8), LLM (5 of 8), Machine Learning (5 of 8), Python (5 of 8)
OTHER ROLES ON THIS TEAM
- Senior Data Platform Engineer posted Nov 17
- Senior AI Engineer posted Nov 9
- Senior ML Engineer, Agentic AI posted Nov 5
- Senior Backend Engineer, Core Services posted Sep 15
- Senior Backend Engineer, Infrastructure posted May 1
- Senior Backend Engineer, Forward Deployed posted May 1
- Senior Full Stack Engineer, Core Services posted May 1
From the team and dates on Ellipsis Health's own job board.
See Backend Engineer salaries in San Francisco →
Similar jobs
- Senior Machine Learning Engineer, Developer AI at ByteDance
Shares 7 skills: Test Automation, Java, TypeScript… · Senior level · same country - Software Engineer (Python, Java, C++, Go, Rust), AI Expert at Weekday AI
Shares 6 skills: Test Automation, Java, Machine Learning… · Senior level · same country - Senior Software Engineer, AI Agents (P345) at 84.51°
Shares 6 skills: Test Automation, TypeScript, JavaScript… · Senior level · same country - Test Engineer III at Heartflow
Shares 7 skills: Test Automation, Java, TypeScript… · Senior level · same city - Senior Software Quality Assurance (SQA) Tester at LTS
Shares 7 skills: Test Automation, Java, TypeScript… · Senior level · same country - Senior AI Engineer at Ellipsis Health
Also at this company
More jobs like this
Is this your posting? Email [email protected] to have it removed.