True Talent Solutions Partners

Research Engineer, Evals

San Francisco
Salary
$250,000 - $290,000 annually
Level
Mid
Work type
On-site · San Francisco
Visa
Sponsors visas

Open. First seen 30 October 2026.

Apply on the company's site →

About the role

About the Role

Build evaluation systems for a web data extraction product, defining how output quality is measured across diverse websites, formats, and edge cases. You will combine research and engineering to turn evaluation results into actionable model and product improvements.

What You'll Do

  • Design rigorous metrics for evaluating structured extraction quality across varied web content.
  • Build scalable evaluation pipelines, testing harnesses, and supporting data systems.
  • Generate and curate ground-truth datasets for trustworthy measurement.
  • Connect evaluation findings to model and product decisions, closing the quality feedback loop.
  • Make evaluation results clear and useful to the broader team.

What We're Looking For

  • Typically 4 to 10 or more years of relevant experience, including at least 2 years working on evaluations.
  • Strong Python skills and experience with machine learning, LLMs, data pipelines, web scraping, NLP, evaluation frameworks, and statistical analysis.
  • Experience building evaluation systems for LLM-based structured extraction from messy, real-world web pages or documents, including field-level accuracy measurement at scale.
  • Experience designing evaluation methods and metrics, building ground-truth datasets, and applying statistical rigor, including sampling, estimator bias and variance, error bounds, and judge-to-human agreement.
  • Ability to combine research methodology with engineering implementation and use evaluation results to guide real model or product decisions.

Compensation & Benefits

Base salary: $250,000 to $290,000 USD annually. Visa sponsorship is available.

Location

Hybrid, based in San Francisco, California. The role requires working on-site five days a week in San Francisco or Toronto, with relocation before starting if needed.

What this role asks for

Particular to this role

NLP

Rare in True Talent Solutions Partners's other 124 open postings.

Standard for software engineering roles

Python

Common in software engineering postings at any company.

Compared with True Talent Solutions Partners's other open jobs.

The team: Client Roles

This role is one of 125 open in Client Roles. 125 were posted in the last 30 days, including 5 lead roles: the team is hiring the people who run it too.

WHAT THE TEAM WORKS WITH, ACROSS ITS POSTINGS

Python (71 of 125), TypeScript (44 of 125), LLM (43 of 125), Machine Learning (36 of 125), React (36 of 125), AWS (28 of 125), JavaScript (24 of 125)

OTHER ROLES ON THIS TEAM

From the team and dates on True Talent Solutions Partners's own job board.

Similar jobs

More jobs like this

SKILLSPython 46,373Machine Learning 19,890LLM 18,671NLP 2,484
PLACESSan Francisco 7,765United States 72,644
COMPANYJobs at True Talent Solutions Partners 125

Is this your posting? Email info@careerholo.com to have it removed.