Machine Learning Engineer, Evaluation & Reliability
- Level
- Mid
- Work type
- Remote · US only
Not stated in the posting: salary, visa.
Open. First seen 12 November 2026.
About the role
About Bask Health
Bask Health's standard text
Bask Health is a software platform that enables entrepreneurs, doctors, and healthcare businesses to launch, manage, and grow direct-to-consumer telehealth businesses.
Our platform brings together patient experiences, clinical workflows, pharmacy operations, and commerce tools in one place.
We're an entrepreneurial, product-focused engineering team that values ownership, technical excellence, and building software that solves real problems. Engineers work closely with product teams and company leadership, take features from idea to production, and have a direct role in shaping the platform.
Engineering at Bask is AI-first and builder-led. We use tools like Cursor and Claude Code throughout development while maintaining high standards for reliability, testing, and software quality.
About the Role
We're looking for a Machine Learning Engineer, Evaluation & Reliability to help ensure that AI-powered features across Bask Health are accurate, dependable, and ready for real-world use.
Bask Health uses software to connect patient experiences, merchant operations, clinical workflows, pharmacy coordination, and commerce capabilities. As AI becomes more integrated into these systems, we need reliable ways to measure model performance, identify failures, and verify that changes improve the user experience without introducing new risks.
You'll help develop the evaluation infrastructure, testing workflows, and performance measurements used to assess large language models, retrieval systems, AI assistants, and automated workflows.
This is a hands-on engineering role that combines machine learning evaluation, software development, data analysis, and production quality assurance. You'll work with model outputs, evaluation datasets, automated testing tools, and application telemetry to understand how AI systems perform under realistic conditions.
Your work may include creating evaluation datasets, implementing automated benchmarks, measuring response quality, testing retrieval accuracy, analyzing model failures, and monitoring production behavior.
You'll collaborate with applied AI engineers, machine learning engineers, backend engineers, product managers, and other stakeholders to define what successful AI behavior looks like for different workflows. You'll help translate those expectations into measurable tests and actionable engineering improvements.
Because Bask operates in healthcare-related environments, AI evaluation must account for accuracy, privacy, appropriate permissions, and the consequences of unreliable responses. You'll help identify situations where AI systems produce unsupported information, expose restricted data, attempt unauthorized actions, or operate outside approved clinical and operational boundaries.
This position emphasizes practical implementation and continuous improvement. You'll contribute to evaluation methodologies and engineering standards while working with senior engineers on broader reliability strategies and more complex technical decisions.
We're looking for an engineer who enjoys investigating how AI systems behave, can turn ambiguous quality problems into measurable tests, and understands that reliable AI products require ongoing evaluation—not just successful demonstrations.
What You'll Do
- Build and maintain evaluation systems for AI-powered applications across Bask Health.
- Develop automated testing frameworks for large language models and AI application workflows.
- Create and maintain representative datasets for evaluating model behavior.
- Implement benchmarks that measure response accuracy, relevance, completeness, and consistency.
- Develop evaluation pipelines for prompts, retrieval systems, and model configurations.
- Test AI assistants and agents against realistic user scenarios and workflow requirements.
- Build automated regression tests to detect performance changes between releases.
- Measure hallucination rates, unsupported claims, and other model quality failures.
- Evaluate retrieval-augmented generation systems for document relevance and answer grounding.
- Develop tests for structured model outputs and schema compliance.
- Evaluate tool-calling accuracy and agent action selection.
- Test AI workflows for unauthorized actions, incorrect permissions, and unsafe behavior.
- Implement evaluation checks for sensitive patient and operational information.
- Build dashboards and reports that communicate AI quality metrics to engineering teams.
- Analyze evaluation results and identify recurring failure patterns.
- Investigate production incidents involving inaccurate or unreliable AI outputs.
- Collaborate with applied AI engineers to improve prompts, retrieval logic, and application behavior.
- Support model comparisons and controlled experiments.
- Evaluate tradeoffs between model quality, latency, reliability, and inference cost.
- Develop automated quality checks for CI/CD and AI feature release workflows.
- Implement monitoring and alerting for production AI performance.
- Build tools for reviewing, labeling, and analyzing model responses.
- Help maintain evaluation datasets as products and user behavior evolve.
- Document evaluation methodologies, test cases, and known model limitations.
- Work with product teams to define measurable acceptance criteria for AI features.
- Collaborate with senior engineers on evaluation standards and reliability improvements.
- Participate in code reviews, technical discussions, and engineering planning.
- Own assigned evaluation and reliability projects from implementation through deployment and ongoing improvement.
Requirements
- 3+ years of professional experience in machine learning engineering, software engineering, applied AI, or a closely related technical field.
- Experience building and maintaining production software or machine learning applications.
- Strong programming skills in Python, TypeScript, or comparable languages.
- Familiarity with large language models and modern generative AI applications.
- Experience testing or evaluating machine learning models or AI-powered software.
- Understanding of machine learning evaluation metrics and experimental methodologies.
- Experience developing automated tests and validation pipelines.
- Familiarity with prompt engineering and model output analysis.
- Ability to analyze structured and unstructured datasets.
- Experience writing scripts or services for data processing and evaluation.
- Understanding of model accuracy, precision, recall, and other performance measures where applicable.
- Familiarity with API integrations and backend application development.
- Experience working with relational databases or analytical data systems.
- Ability to identify patterns in model failures and translate findings into engineering improvements.
- Understanding of regression testing and software quality assurance.
- Familiarity with logging, monitoring, and production troubleshooting.
- Experience using Git and collaborative software development workflows.
- Understanding of the limitations and failure modes of large language models.
- Ability to design representative test cases and evaluation scenarios.
- Familiarity with cloud-based development and deployment environments.
- Experience writing maintainable, well-tested production code.
- Strong analytical thinking and attention to detail.
- Ability to communicate technical evaluation results clearly.
- Experience collaborating with engineers, product managers, and cross-functional teams.
- Ability to independently own assigned technical projects.
Nice to Have
- Experience building evaluation frameworks for large language models.
- Familiarity with OpenAI, Anthropic, or comparable model APIs.
- Experience with tools such as LangSmith, DeepEval, Ragas, or comparable AI evaluation frameworks.
- Familiarity with LLM-as-a-judge evaluation techniques.
- Experience creating human-labeled evaluation datasets.
- Understanding of inter-rater agreement and evaluation consistency.
- Experience evaluating retrieval-augmented generation systems.
- Familiarity with embeddings, semantic search, and vector databases.
- Experience evaluating AI agents and tool-calling workflows.
- Familiarity with prompt injection testing and AI security evaluations.
- Experience implementing automated regression testing for AI applications.
- Familiarity with experiment tracking and model comparison tools.
- Experience building dashboards for machine learning performance monitoring.
- Understanding of statistical analysis and experiment design.
- Experience working with Python data analysis libraries.
- Familiarity with TypeScript and Node.js application development.
- Experience working with AWS or serverless infrastructure.
- Experience evaluating AI systems used in healthcare or other sensitive environments.
- Understanding of HIPAA and healthcare data privacy considerations.
- Experience using AI-assisted development tools such as Cursor or Claude Code.
What this role asks for
Particular to this role
CI/CD · Generative AI · Prompt Engineering · RAG · Test Automation · Vector Databases
Rare in Bask Health's other 203 open postings.
Standard for data science / ML roles
LLM · Machine Learning · Python
Common in data science / ML postings at any company.
97% of this description is written for the role. The rest is Bask Health's standard text, folded away below.
Compared with Bask Health's other open jobs.
The team: Engineering HIRES STEADILY
This role is one of 77 open in Engineering. 70 were posted in the last 30 days and the oldest in December: a team that hires all year rather than one being built.
WHAT THE TEAM WORKS WITH, ACROSS ITS POSTINGS
TypeScript (74 of 77), JavaScript (70 of 77), Next.js (63 of 77), React (63 of 77), AWS (58 of 77), Node.js (58 of 77), Serverless (58 of 77)
OTHER ROLES ON THIS TEAM
- Senior Applied AI Engineer, Patient Experience posted Nov 12
- AI Engineer, Merchant Copilots posted Nov 12
- Applied ML Engineer, Agents & Automation posted Nov 12
- Machine Learning Engineer, Personalization posted Nov 12
- Machine Learning Engineering Intern posted Nov 12
- Frontend Engineer, Patient Experience & Intakes posted Nov 12
- Forward Deployed Engineer, Pharmacy Integrations posted Nov 12
- Forward Deployed Product Engineer posted Nov 12
- Frontend Engineer, EMR & Clinical Workflows posted Nov 12
- Frontend Engineer, New Grad posted Nov 12
- Junior Forward Deployed Engineer posted Nov 12
- Creative Frontend Developer posted Nov 12
- Frontend Engineer, Sites & Studio posted Nov 12
- Frontend Engineering Intern posted Nov 12
- Head of Applied AI posted Nov 12
- Creative Engineering Intern posted Nov 12
- Design Engineer, WebGL & Three.js posted Nov 12
- Forward Deployed Engineer posted Nov 12
- Forward Deployed Engineer, AI & Automation posted Nov 12
- Forward Deployed Engineer, AI Solutions posted Nov 12
- Forward Deployed Engineer, Merchant Solutions posted Nov 12
- Junior Design Engineer posted Nov 12
- Senior Machine Learning Engineer, Evaluation & Reliability posted Nov 12
- Senior Machine Learning Engineer, Search & Personalization posted Nov 12
- ML Platform Engineer, Evaluation Systems posted Nov 12
- Machine Learning Engineer, Search & Discovery posted Nov 12
- AI Engineering Intern posted Nov 12
- Applied AI Engineer, Patient Experience posted Nov 12
- Applied ML Engineer, Care Experience posted Nov 12
- Software Engineer, AI Platform posted Nov 12
- Deployment Engineer, Merchant Solutions posted Nov 12
- ML Systems Engineer posted Nov 12
- AI Evaluation Engineer posted Nov 12
- AI Product Engineer, Clinical & Pharmacy Workflows posted Nov 12
- Junior Applied AI Engineer posted Nov 12
- Software Engineer, AI Quality & Evaluation posted Nov 12
- Junior Backend Engineer posted Nov 11
- Senior Forward Deployed Engineer, AI & Automation posted Nov 11
- Senior Forward Deployed Engineer, Merchant Solutions posted Nov 11
- Senior Frontend Engineer, Sites & Studio posted Nov 11
- Senior Full Stack Product Engineer, Merchant Platform posted Nov 11
- Senior Platform Engineer, APIs & Workflow Automation posted Nov 11
- Backend Engineer, Care Platform & EMR posted Nov 11
- Backend Engineer, Pharmacy Integrations posted Nov 11
- Backend Engineering Intern posted Nov 11
- Full Stack Product Engineer, Merchant Platform posted Nov 11
- Junior Product Engineer, Interactive Experiences posted Nov 11
- Platform Engineer, APIs & Workflow Automation posted Nov 11
- Product Engineer, Merchant Portal & Commerce posted Nov 11
- Product Engineer, New Grad posted Nov 11
- Software Engineer, Early Career posted Nov 11
- Product Engineering Intern posted Nov 11
- Senior Backend Engineer, Pharmacy Integrations posted Nov 11
- Senior Frontend Engineer, Patient Experience & Intakes posted Nov 11
- Software Engineering Intern posted Nov 11
- Backend Engineer, Billing & Revenue Systems posted Nov 11
- Junior Full Stack Engineer, Merchant Tools posted Nov 11
- Backend Engineer, New Grad posted Nov 11
- Junior Full Stack Engineer posted Nov 11
- Senior Backend Engineer, Billing & Revenue Systems posted Nov 11
- Senior Frontend Engineer, EMR & Clinical Workflows posted Nov 11
- Junior Backend Engineer, Platform posted Nov 11
- Senior Backend Engineer, Care Platform & EMR posted Nov 11
- Senior Design Engineer, WebGL & Three.js posted Nov 11
- Front-End Software Engineer (Remote) posted Nov 2
- Front-End Software Engineer posted Nov 2
- Forward-Deployed Engineer posted Oct 30
- Junior Back-End Software Engineer posted Oct 27
- Junior Front-End Software Engineer posted Oct 27
- Product Owner posted Aug 31
- Senior Front-End Software Engineer (Remote) posted Aug 20
- Three.js Front End Engineer posted Aug 19
- Product Manager Egypt posted Jul 14
- Senior Back-End Software Engineer posted Dec 13
- Senior Back-End Software Engineer (Remote) posted Dec 13
- Back-End Software Engineer posted Dec 13
From the team and dates on Bask Health's own job board.
See Machine Learning Engineer salaries in the United States →
Similar jobs
- Senior Forward Deployed Engineer (Gemini Enterprise) at Welcome to the Jungle
Shares 13 skills: Serverless, Vector Databases, RAG… - (1119604) Lead Software Engineer - Emerging Technologies at Diversified Services Network, Inc.
Shares 11 skills: Serverless, Vector Databases, RAG… · same country - Software Engineer II, Fintech at Twitch
Shares 10 skills: Serverless, Vector Databases, RAG… · same seniority · same country - Senior Software Engineer (AI & Full Stack) at Fable
Shares 12 skills: Serverless, Vector Databases, Prompt Engineering… - AI Web Developer T4 - Active TS/SCI with Polygraph at TOMORROW HIRE
Shares 8 skills: Serverless, Prompt Engineering, RAG… · same seniority · same country - Solutions Engineering Intern at Bask Health
Also at this company
More jobs like this
Is this your posting? Email [email protected] to have it removed.