AI Evaluation Engineer
- Level
- Mid
- Work type
- Remote · US only
Not stated in the posting: salary, visa.
Open. First seen 12 November 2026.
About the role
About Bask Health
Bask Health's standard text
Bask Health is a software platform that enables entrepreneurs, doctors, and healthcare businesses to launch, manage, and grow direct-to-consumer telehealth businesses.
Our platform brings together patient experiences, clinical workflows, pharmacy operations, and commerce tools in one place.
We're an entrepreneurial, product-focused engineering team that values ownership, technical excellence, and building software that solves real problems. Engineers work closely with product teams and company leadership, take features from idea to production, and have a direct role in shaping the platform.
Engineering at Bask is AI-first and builder-led. We use tools like Cursor and Claude Code throughout development while maintaining high standards for reliability, testing, and software quality.
About the Role
We're looking for an AI Evaluation Engineer to help ensure that AI-powered features across Bask Health behave accurately, reliably, and consistently before and after they reach production.
Bask Health supports the technology behind direct-to-consumer telehealth businesses, connecting patient experiences, merchant operations, clinical workflows, pharmacy coordination, and commerce capabilities. As we introduce more AI-powered functionality, we need dependable ways to verify that these systems work as intended under realistic conditions.
You'll build and maintain evaluation workflows that test large language models, AI assistants, retrieval systems, and automated features against clearly defined product requirements.
This is a hands-on engineering role that combines AI evaluation, software testing, automation, and production quality assurance. You'll develop test scenarios, evaluation datasets, automated checks, and tools that help engineering teams identify problems before they affect users.
Your work may include testing AI-generated responses, validating structured outputs, evaluating tool-calling behavior, checking retrieval accuracy, and identifying regressions caused by changes to prompts, models, or application code.
You'll collaborate with applied AI engineers, machine learning engineers, backend engineers, product managers, and quality-focused engineering teams to define expected behavior and translate those expectations into repeatable evaluations.
You'll also help establish processes for reviewing AI failures, documenting recurring issues, and determining whether a feature meets its acceptance criteria. You'll work with engineers to reproduce problems and verify that fixes address the underlying issue.
Because Bask operates in healthcare-related environments, you'll pay particular attention to privacy, permissions, unsupported medical claims, and AI behavior that could interfere with approved clinical or pharmacy workflows. Evaluation systems should help verify that AI features remain within their intended operational boundaries.
This position emphasizes practical testing, thoughtful engineering, and continuous improvement. You'll contribute to evaluation standards while independently implementing and maintaining the tools and tests that support AI quality.
We're looking for an engineer who enjoys finding difficult edge cases, understands how AI behavior differs from traditional deterministic software, and wants to help build products that users can depend on.
What You'll Do
- Design and implement evaluation workflows for AI-powered applications across Bask Health.
- Build automated tests for large language model outputs and AI application behavior.
- Develop evaluation datasets that represent realistic user requests and product workflows.
- Create test scenarios for patient-facing, merchant-facing, and internal AI features.
- Implement checks for response accuracy, relevance, completeness, and consistency.
- Evaluate AI-generated content for unsupported claims and misleading information.
- Test structured model outputs for schema compliance and application compatibility.
- Build evaluations for AI tool calling and authorized action execution.
- Validate that AI assistants respect application permissions and tenant boundaries.
- Test retrieval-augmented generation systems for grounding and information relevance.
- Develop regression tests that identify unexpected changes in AI behavior.
- Integrate evaluation checks into engineering development and release workflows.
- Create automated comparison tests for prompts, model versions, and configuration changes.
- Develop scripts and utilities that simplify AI evaluation execution.
- Build reporting tools that summarize evaluation results and failure patterns.
- Investigate failed evaluations and help engineers reproduce underlying issues.
- Work with product managers to define expected AI behavior and acceptance criteria.
- Collaborate with applied AI engineers to improve prompts and application logic.
- Test AI applications against unusual inputs, ambiguous requests, and difficult edge cases.
- Develop evaluation cases for prompt injection and unauthorized tool-use attempts.
- Support human review workflows for outputs that require subjective assessment.
- Track recurring AI failures and help prioritize reliability improvements.
- Monitor production evaluation signals and investigate quality regressions.
- Document test coverage, evaluation procedures, and known limitations.
- Improve evaluation efficiency, repeatability, and maintainability.
- Participate in code reviews, engineering planning, and technical discussions.
- Support controlled testing and rollout of new AI-powered features.
- Own assigned AI evaluation projects from test design through implementation, reporting, and ongoing improvement.
Requirements
- 3+ years of professional experience in software engineering, AI engineering, test automation, quality engineering, or a closely related technical field.
- Experience developing automated tests for production software applications.
- Strong programming skills in Python, TypeScript, JavaScript, or comparable languages.
- Familiarity with large language models and modern AI-powered applications.
- Understanding of software testing methodologies and test case design.
- Experience building automated testing scripts, frameworks, or validation tools.
- Familiarity with API testing and backend application integrations.
- Understanding of prompt engineering and common AI model failure modes.
- Ability to evaluate AI-generated responses against expected requirements.
- Experience working with structured data and validating application outputs.
- Familiarity with regression testing and software release processes.
- Experience debugging application failures and reproducing technical issues.
- Understanding of authentication, authorization, and secure application behavior.
- Familiarity with databases and application data models.
- Experience working with Git and collaborative software development workflows.
- Understanding of continuous integration and automated testing pipelines.
- Familiarity with logging, monitoring, and production troubleshooting.
- Ability to create clear, reproducible test cases.
- Strong analytical thinking and attention to detail.
- Ability to distinguish application defects from model behavior limitations.
- Experience documenting technical issues and communicating findings.
- Familiarity with cloud-based application environments.
- Ability to collaborate effectively with engineers, product managers, and other stakeholders.
- Strong problem-solving skills and a quality-focused engineering mindset.
- Ability to independently own evaluation and test automation projects.
Nice to Have
- Experience evaluating large language models or generative AI applications.
- Familiarity with OpenAI, Anthropic, or comparable model APIs.
- Experience with LangSmith, DeepEval, Ragas, promptfoo, or comparable evaluation tools.
- Familiarity with LLM-as-a-judge evaluation techniques.
- Experience developing AI evaluation datasets and test benchmarks.
- Understanding of human evaluation and response labeling workflows.
- Experience testing retrieval-augmented generation systems.
- Familiarity with embeddings, semantic search, and vector databases.
- Experience evaluating AI agents and tool-calling behavior.
- Familiarity with prompt injection testing and AI security evaluations.
- Experience implementing AI regression tests in CI/CD pipelines.
- Experience building automated evaluation reports or dashboards.
- Familiarity with Playwright, Jest, Vitest, Pytest, or comparable testing frameworks.
- Experience testing TypeScript or Node.js applications.
- Familiarity with React or Next.js application testing.
- Experience working with AWS or serverless infrastructure.
- Experience evaluating AI features in healthcare or other sensitive environments.
- Understanding of HIPAA and healthcare data privacy considerations.
- Experience developing human-in-the-loop quality review processes.
- Experience using AI-assisted development tools such as Cursor or Claude Code.
What this role asks for
Particular to this role
CI/CD · Generative AI · Jest · Machine Learning · Playwright · Prompt Engineering · Pytest · RAG
Rare in Bask Health's other 203 open postings.
Standard for software engineering roles
Python
Common in software engineering postings at any company.
97% of this description is written for the role. The rest is Bask Health's standard text, folded away below.
Compared with Bask Health's other open jobs.
The team: Engineering HIRES STEADILY
This role is one of 77 open in Engineering. 70 were posted in the last 30 days and the oldest in December: a team that hires all year rather than one being built.
WHAT THE TEAM WORKS WITH, ACROSS ITS POSTINGS
TypeScript (74 of 77), JavaScript (70 of 77), Next.js (63 of 77), React (63 of 77), AWS (58 of 77), Node.js (58 of 77), Serverless (58 of 77)
OTHER ROLES ON THIS TEAM
- Senior Applied AI Engineer, Patient Experience posted Nov 12
- AI Engineer, Merchant Copilots posted Nov 12
- Applied ML Engineer, Agents & Automation posted Nov 12
- Machine Learning Engineer, Personalization posted Nov 12
- Machine Learning Engineering Intern posted Nov 12
- Frontend Engineer, Patient Experience & Intakes posted Nov 12
- Forward Deployed Engineer, Pharmacy Integrations posted Nov 12
- Forward Deployed Product Engineer posted Nov 12
- Frontend Engineer, EMR & Clinical Workflows posted Nov 12
- Frontend Engineer, New Grad posted Nov 12
- Junior Forward Deployed Engineer posted Nov 12
- Creative Frontend Developer posted Nov 12
- Frontend Engineer, Sites & Studio posted Nov 12
- Frontend Engineering Intern posted Nov 12
- Head of Applied AI posted Nov 12
- Creative Engineering Intern posted Nov 12
- Design Engineer, WebGL & Three.js posted Nov 12
- Forward Deployed Engineer posted Nov 12
- Forward Deployed Engineer, AI & Automation posted Nov 12
- Forward Deployed Engineer, AI Solutions posted Nov 12
- Forward Deployed Engineer, Merchant Solutions posted Nov 12
- Junior Design Engineer posted Nov 12
- Senior Machine Learning Engineer, Evaluation & Reliability posted Nov 12
- Senior Machine Learning Engineer, Search & Personalization posted Nov 12
- ML Platform Engineer, Evaluation Systems posted Nov 12
- Machine Learning Engineer, Search & Discovery posted Nov 12
- AI Engineering Intern posted Nov 12
- Applied AI Engineer, Patient Experience posted Nov 12
- Applied ML Engineer, Care Experience posted Nov 12
- Software Engineer, AI Platform posted Nov 12
- Deployment Engineer, Merchant Solutions posted Nov 12
- Machine Learning Engineer, Evaluation & Reliability posted Nov 12
- ML Systems Engineer posted Nov 12
- AI Product Engineer, Clinical & Pharmacy Workflows posted Nov 12
- Junior Applied AI Engineer posted Nov 12
- Software Engineer, AI Quality & Evaluation posted Nov 12
- Junior Backend Engineer posted Nov 11
- Senior Forward Deployed Engineer, AI & Automation posted Nov 11
- Senior Forward Deployed Engineer, Merchant Solutions posted Nov 11
- Senior Frontend Engineer, Sites & Studio posted Nov 11
- Senior Full Stack Product Engineer, Merchant Platform posted Nov 11
- Senior Platform Engineer, APIs & Workflow Automation posted Nov 11
- Backend Engineer, Care Platform & EMR posted Nov 11
- Backend Engineer, Pharmacy Integrations posted Nov 11
- Backend Engineering Intern posted Nov 11
- Full Stack Product Engineer, Merchant Platform posted Nov 11
- Junior Product Engineer, Interactive Experiences posted Nov 11
- Platform Engineer, APIs & Workflow Automation posted Nov 11
- Product Engineer, Merchant Portal & Commerce posted Nov 11
- Product Engineer, New Grad posted Nov 11
- Software Engineer, Early Career posted Nov 11
- Product Engineering Intern posted Nov 11
- Senior Backend Engineer, Pharmacy Integrations posted Nov 11
- Senior Frontend Engineer, Patient Experience & Intakes posted Nov 11
- Software Engineering Intern posted Nov 11
- Backend Engineer, Billing & Revenue Systems posted Nov 11
- Junior Full Stack Engineer, Merchant Tools posted Nov 11
- Backend Engineer, New Grad posted Nov 11
- Junior Full Stack Engineer posted Nov 11
- Senior Backend Engineer, Billing & Revenue Systems posted Nov 11
- Senior Frontend Engineer, EMR & Clinical Workflows posted Nov 11
- Junior Backend Engineer, Platform posted Nov 11
- Senior Backend Engineer, Care Platform & EMR posted Nov 11
- Senior Design Engineer, WebGL & Three.js posted Nov 11
- Front-End Software Engineer (Remote) posted Nov 2
- Front-End Software Engineer posted Nov 2
- Forward-Deployed Engineer posted Oct 30
- Junior Back-End Software Engineer posted Oct 27
- Junior Front-End Software Engineer posted Oct 27
- Product Owner posted Aug 31
- Senior Front-End Software Engineer (Remote) posted Aug 20
- Three.js Front End Engineer posted Aug 19
- Product Manager Egypt posted Jul 14
- Senior Back-End Software Engineer posted Dec 13
- Senior Back-End Software Engineer (Remote) posted Dec 13
- Back-End Software Engineer posted Dec 13
From the team and dates on Bask Health's own job board.
Similar jobs
- Senior Software Engineer, AI Agents (P345) at 84.51°
Shares 14 skills: Pytest, Jest, Playwright… · same country - AI Developer at Valsoft Corporation
Shares 14 skills: Jest, Playwright, Next.js… · same seniority - Full Stack Developer at Virallens
Shares 15 skills: Jest, Playwright, Next.js… · same seniority - Senior QA Engineer - AI Products (6 Months Contract) at Arabic.AI
Shares 14 skills: Pytest, Playwright, Vector Databases… - Part time Sr. Software Engineer at GeoVisual Analytics
Shares 14 skills: Pytest, Jest, Next.js… · same remote region - Solutions Engineering Intern at Bask Health
Also at this company
More jobs like this
Is this your posting? Email [email protected] to have it removed.