AI Evals/Benchmarking Engineer - A26419

We believe in positively impacting the lives of people around us and the environment we live in through the use of technology. Hence, we are committed to providing a conducive environment for all employees to realise their full potential, who in turn have the opportunity to continuously drive innovation.

Co-Development Business Unit is hiring for AI Evals/Benchmarking Engineer

You will work in Singapore Government Agencies

This is a fixed term contract role. The engagement is 1 year.

Internal Code: A26419

What will you do?

• Design and operate the evaluation and benchmarking framework for the AIDD target evaluation workflow and its AI agents. The role establishes retrospective baselines, integrates Therapeutics Data Commons (TDC) datasets where appropriate, runs forward regression testing, and converts structured human-in-the-loop feedback into governed training and improvement signals while maintaining quality, provenance and reproducibility.
• Define the end-to-end evaluation strategy, benchmark suites, test datasets, rubrics, acceptance thresholds and reporting standards for retrieval, generation, reasoning, scoring, gating and meta-agent components.
• Build retrospective benchmarks using historical target evaluations and expert decisions to measure evidence quality, citation and provenance coverage, factual consistency, therapeutic-direction accuracy, score reliability and report quality.
• Assess and integrate relevant TDC datasets and tasks, including data mapping, licensing, preprocessing, versioning and alignment with AIDD evaluation criteria.
• Develop automated forward regression pipelines that compare models, prompts, agents, data sources and workflow versions, detect performance drift and block unsafe or substandard releases.
• Design human-in-the-loop review capture, including structured annotations, corrections, confidence, rationale and outcomes, and transform validated feedback into governed evaluation and training signals.
• Perform error analysis and failure-mode discovery, maintain traceable experiment records, and work with scientists and engineers to prioritize and verify improvements.
• Create dashboards and concise reports on benchmark performance, QA pass rate, provenance, latency, cost and human-review effort. Review results with scientific stakeholders, the Architect and the Delivery & Tech Lead to support model selection, release gates and prioritisation of workflow and engineering improvements.

Requirements

What are we looking for?

• Strong experience in AI or machine-learning engineering, model evaluation or applied research, including generative AI, LLMs, RAG or agentic systems.
• Proficiency in Python and common ML and data libraries, with the ability to build automated data, evaluation and reporting pipelines.
• Practical knowledge of evaluation methods for retrieval, generation, classification, ranking and scoring, including metric selection, dataset design, statistical comparison and error analysis.
• Experience with experiment tracking, data and model versioning, reproducibility and MLOps practices using tools such as MLflow, DVC, Airflow, Git and containers.
• Ability to analyse false positives, false negatives, edge cases, drift and failure modes, then translate findings into actionable engineering recommendations.
• Strong documentation and stakeholder communication skills, with the ability to work closely with domain experts, AI engineers and governance teams.
• Relevant degree in artificial intelligence, computer science, data science, engineering or a related discipline.
• Experience with TDC or comparable biomedical datasets and evaluation of RAG, semantic search, document intelligence or LLM agents.
• Development of multi-agent evaluation systems, human-feedback datasets and traceable scoring or decision logic.
• Production AI delivery on cloud and data platforms, including containerised infrastructure, MLOps and evaluation or diagnostic tooling.

Benefits

What do we offer in return?

Competitive Compensation: Market competitive salary and variable performance bonus aligned to your skills, impact, and contribution.

Why you'll love working with us?

Does it sound like something you are interested in exploring further? Please be in touch with our team for an initial chat at careers@activate.sg