Software Engineer, AI Quality & Evaluation
- Level
- Mid
- Work type
- Remote · US only
Not stated in the posting: salary, visa.
Open. First seen 12 November 2026.
About the role
About Bask Health
Bask Health's standard text
Bask Health is a software platform that enables entrepreneurs, doctors, and healthcare businesses to launch, manage, and grow direct-to-consumer telehealth businesses.
Our platform brings together patient experiences, clinical workflows, pharmacy operations, and commerce tools in one place.
We're an entrepreneurial, product-focused engineering team that values ownership, technical excellence, and building software that solves real problems. Engineers work closely with product teams and company leadership, take features from idea to production, and have a direct role in shaping the platform.
Engineering at Bask is AI-first and builder-led. We use tools like Cursor and Claude Code throughout development while maintaining high standards for reliability, testing, and software quality.
About the Role
We're looking for a Software Engineer, AI Quality & Evaluation to build the engineering systems that help Bask Health test, evaluate, and improve AI-powered applications throughout their development and production lifecycles.
Bask Health supports direct-to-consumer telehealth businesses through a platform that connects patient experiences, merchant operations, clinical workflows, pharmacy coordination, and commerce tools. As we introduce more AI-powered capabilities across these areas, we need reliable software infrastructure that makes it easy for engineering teams to understand how those systems behave.
You'll help develop the internal tools, backend services, testing frameworks, and automation that support AI quality assurance across the platform.
This is a hands-on software engineering role focused on building reusable systems rather than evaluating individual models in isolation. You'll work with APIs, databases, automated testing infrastructure, model integrations, and developer tools to make AI evaluation more efficient, consistent, and accessible.
Your work may include evaluation execution services, test dataset management tools, automated regression pipelines, result comparison systems, and interfaces that help engineers investigate AI failures.
You'll collaborate with applied AI engineers, machine learning engineers, AI evaluation specialists, and backend engineers to understand their testing needs and translate those requirements into dependable software capabilities.
You'll also help integrate AI evaluation into normal engineering workflows. This includes making it easier to run tests during development, compare model or prompt changes, inspect failures, and identify regressions before production releases.
Because Bask operates in healthcare-related environments, you'll help ensure evaluation systems handle sensitive information appropriately, respect access permissions, and support testing for privacy, safety, and approved operational boundaries.
This position emphasizes strong software engineering fundamentals, developer experience, and production reliability. You'll contribute to technical architecture, write maintainable code, improve infrastructure performance, and own services that other engineering teams rely on.
We're looking for an engineer who enjoys building tools for other developers, understands the challenges of testing nondeterministic AI systems, and wants to create reliable infrastructure that improves software quality across the organization.
What You'll Do
- Design, build, and maintain software infrastructure that supports AI quality and evaluation across Bask Health.
- Develop backend services that execute and manage automated AI evaluations.
- Build reusable testing tools for large language model applications.
- Create APIs that allow engineering teams to submit evaluation jobs and retrieve results.
- Develop systems for managing evaluation datasets, test cases, and expected outcomes.
- Build automated workflows for comparing prompts, models, and application configurations.
- Implement regression testing infrastructure for AI-powered features.
- Integrate AI evaluation checks into continuous integration and deployment pipelines.
- Develop services that record and organize evaluation results over time.
- Build tools that help engineers inspect AI responses, failures, and test histories.
- Implement structured result storage and comparison capabilities.
- Develop internal dashboards or interfaces for reviewing evaluation outcomes.
- Create reusable software libraries that simplify AI testing across multiple applications.
- Build systems that support both deterministic checks and model-based evaluations.
- Implement automated testing for tool-calling behavior and structured model outputs.
- Develop infrastructure for evaluating retrieval-augmented generation applications.
- Build asynchronous processing systems for large evaluation workloads.
- Implement job scheduling, retries, timeouts, and error handling for evaluation services.
- Develop monitoring and logging for evaluation infrastructure.
- Optimize evaluation execution speed, reliability, and infrastructure costs.
- Work with AI evaluation engineers to translate testing requirements into reusable software tools.
- Collaborate with machine learning engineers to support model and prompt comparison workflows.
- Implement appropriate access controls and secure handling of evaluation data.
- Ensure evaluation infrastructure respects tenant boundaries and data privacy requirements.
- Investigate infrastructure failures and resolve issues affecting evaluation reliability.
- Improve developer documentation and internal tooling for AI testing.
- Participate in code reviews, technical planning, and architecture discussions.
- Own assigned AI quality engineering projects from design through deployment, monitoring, and continuous improvement.
Requirements
- 3+ years of professional software engineering experience, preferably in backend, platform, infrastructure, or developer tooling roles.
- Experience building and maintaining production backend services.
- Strong programming skills in TypeScript, JavaScript, Python, or comparable languages.
- Experience designing and developing APIs and service integrations.
- Familiarity with automated testing frameworks and software quality practices.
- Understanding of continuous integration and deployment workflows.
- Experience building reusable software components or internal engineering tools.
- Familiarity with large language models and modern AI application development.
- Understanding of the challenges involved in testing nondeterministic AI systems.
- Experience working with relational databases and structured application data.
- Familiarity with asynchronous processing and background job systems.
- Understanding of distributed systems fundamentals and service reliability.
- Experience developing automated testing pipelines.
- Familiarity with cloud-based application deployment.
- Understanding of authentication, authorization, and secure service communication.
- Experience implementing logging, monitoring, and production diagnostics.
- Ability to troubleshoot backend services and infrastructure issues.
- Familiarity with configuration management and software release processes.
- Experience writing maintainable, well-tested production code.
- Understanding of API error handling, retries, and failure recovery.
- Experience working with Git and collaborative engineering workflows.
- Ability to translate technical requirements into reusable engineering systems.
- Strong analytical thinking and problem-solving skills.
- Ability to collaborate effectively with software engineers, AI engineers, and machine learning specialists.
- Ability to independently own engineering projects from technical design through deployment.
Nice to Have
- Experience building infrastructure for AI evaluation or machine learning testing.
- Familiarity with OpenAI, Anthropic, or comparable model APIs.
- Experience with LangSmith, DeepEval, Ragas, promptfoo, or comparable evaluation tools.
- Familiarity with LLM-as-a-judge evaluation approaches.
- Experience building evaluation execution services or automated benchmark systems.
- Familiarity with AI regression testing and prompt comparison workflows.
- Experience developing internal developer platforms or SDKs.
- Experience building testing infrastructure for distributed applications.
- Familiarity with TypeScript and Node.js backend development.
- Experience with React or Next.js for internal tooling interfaces.
- Familiarity with tRPC, Prisma, Drizzle, or comparable development tools.
- Experience working with AWS Lambda, SQS, EventBridge, or related cloud services.
- Familiarity with serverless infrastructure and infrastructure-as-code tools.
- Experience with SST or comparable application deployment frameworks.
- Experience building scalable asynchronous job processing systems.
- Familiarity with embeddings, vector databases, and retrieval-augmented generation.
- Experience implementing model observability or AI request tracing.
- Understanding of multi-tenant SaaS architecture and data isolation.
- Familiarity with HIPAA and healthcare data privacy considerations.
- Experience using AI-assisted development tools such as Cursor or Claude Code.
What this role asks for
Particular to this role
Machine Learning · RAG · Test Automation · Vector Databases
Rare in Bask Health's other 203 open postings.
Standard for software engineering roles
Python
Common in software engineering postings at any company.
97% of this description is written for the role. The rest is Bask Health's standard text, folded away below.
Compared with Bask Health's other open jobs.
The team: Engineering HIRES STEADILY
This role is one of 77 open in Engineering. 70 were posted in the last 30 days and the oldest in December: a team that hires all year rather than one being built.
WHAT THE TEAM WORKS WITH, ACROSS ITS POSTINGS
TypeScript (74 of 77), JavaScript (70 of 77), Next.js (63 of 77), React (63 of 77), AWS (58 of 77), Node.js (58 of 77), Serverless (58 of 77)
OTHER ROLES ON THIS TEAM
- Senior Applied AI Engineer, Patient Experience posted Nov 12
- AI Engineer, Merchant Copilots posted Nov 12
- Applied ML Engineer, Agents & Automation posted Nov 12
- Machine Learning Engineer, Personalization posted Nov 12
- Machine Learning Engineering Intern posted Nov 12
- Frontend Engineer, Patient Experience & Intakes posted Nov 12
- Forward Deployed Engineer, Pharmacy Integrations posted Nov 12
- Forward Deployed Product Engineer posted Nov 12
- Frontend Engineer, EMR & Clinical Workflows posted Nov 12
- Frontend Engineer, New Grad posted Nov 12
- Junior Forward Deployed Engineer posted Nov 12
- Creative Frontend Developer posted Nov 12
- Frontend Engineer, Sites & Studio posted Nov 12
- Frontend Engineering Intern posted Nov 12
- Head of Applied AI posted Nov 12
- Creative Engineering Intern posted Nov 12
- Design Engineer, WebGL & Three.js posted Nov 12
- Forward Deployed Engineer posted Nov 12
- Forward Deployed Engineer, AI & Automation posted Nov 12
- Forward Deployed Engineer, AI Solutions posted Nov 12
- Forward Deployed Engineer, Merchant Solutions posted Nov 12
- Junior Design Engineer posted Nov 12
- Senior Machine Learning Engineer, Evaluation & Reliability posted Nov 12
- Senior Machine Learning Engineer, Search & Personalization posted Nov 12
- ML Platform Engineer, Evaluation Systems posted Nov 12
- Machine Learning Engineer, Search & Discovery posted Nov 12
- AI Engineering Intern posted Nov 12
- Applied AI Engineer, Patient Experience posted Nov 12
- Applied ML Engineer, Care Experience posted Nov 12
- Software Engineer, AI Platform posted Nov 12
- Deployment Engineer, Merchant Solutions posted Nov 12
- Machine Learning Engineer, Evaluation & Reliability posted Nov 12
- ML Systems Engineer posted Nov 12
- AI Evaluation Engineer posted Nov 12
- AI Product Engineer, Clinical & Pharmacy Workflows posted Nov 12
- Junior Applied AI Engineer posted Nov 12
- Junior Backend Engineer posted Nov 11
- Senior Forward Deployed Engineer, AI & Automation posted Nov 11
- Senior Forward Deployed Engineer, Merchant Solutions posted Nov 11
- Senior Frontend Engineer, Sites & Studio posted Nov 11
- Senior Full Stack Product Engineer, Merchant Platform posted Nov 11
- Senior Platform Engineer, APIs & Workflow Automation posted Nov 11
- Backend Engineer, Care Platform & EMR posted Nov 11
- Backend Engineer, Pharmacy Integrations posted Nov 11
- Backend Engineering Intern posted Nov 11
- Full Stack Product Engineer, Merchant Platform posted Nov 11
- Junior Product Engineer, Interactive Experiences posted Nov 11
- Platform Engineer, APIs & Workflow Automation posted Nov 11
- Product Engineer, Merchant Portal & Commerce posted Nov 11
- Product Engineer, New Grad posted Nov 11
- Software Engineer, Early Career posted Nov 11
- Product Engineering Intern posted Nov 11
- Senior Backend Engineer, Pharmacy Integrations posted Nov 11
- Senior Frontend Engineer, Patient Experience & Intakes posted Nov 11
- Software Engineering Intern posted Nov 11
- Backend Engineer, Billing & Revenue Systems posted Nov 11
- Junior Full Stack Engineer, Merchant Tools posted Nov 11
- Backend Engineer, New Grad posted Nov 11
- Junior Full Stack Engineer posted Nov 11
- Senior Backend Engineer, Billing & Revenue Systems posted Nov 11
- Senior Frontend Engineer, EMR & Clinical Workflows posted Nov 11
- Junior Backend Engineer, Platform posted Nov 11
- Senior Backend Engineer, Care Platform & EMR posted Nov 11
- Senior Design Engineer, WebGL & Three.js posted Nov 11
- Front-End Software Engineer (Remote) posted Nov 2
- Front-End Software Engineer posted Nov 2
- Forward-Deployed Engineer posted Oct 30
- Junior Back-End Software Engineer posted Oct 27
- Junior Front-End Software Engineer posted Oct 27
- Product Owner posted Aug 31
- Senior Front-End Software Engineer (Remote) posted Aug 20
- Three.js Front End Engineer posted Aug 19
- Product Manager Egypt posted Jul 14
- Senior Back-End Software Engineer posted Dec 13
- Senior Back-End Software Engineer (Remote) posted Dec 13
- Back-End Software Engineer posted Dec 13
From the team and dates on Bask Health's own job board.
See Software Engineer salaries in the United States →
Similar jobs
- Full-Stack AI Engineer at Pavago
Shares 12 skills: Serverless, Next.js, Vector Databases… · same seniority · same remote region - Python / AI Solutions Full-Stack Developer at Total Performance Consulting
Shares 12 skills: Serverless, Next.js, Vector Databases… · same seniority - Front End AI Engineer at G-P
Shares 10 skills: Serverless, Next.js, Vector Databases… · same seniority · same remote region - Software Engineer - Platform / Applied AI (Full-Stack) at JeffreyM Consulting
Shares 8 skills: Serverless, Next.js, Node.js… · same seniority · same country - AI-Native Full-Stack Software Engineer / Technical Project Manager at Zenara Health
Shares 10 skills: Serverless, Next.js, Vector Databases… - Solutions Engineering Intern at Bask Health
Also at this company
More jobs like this
Is this your posting? Email [email protected] to have it removed.