True Talent Solutions Partners

Research Engineer, Evals

San Francisco
Salary
$250,000 - $290,000 annually
Level
Mid
Work type
On-site · San Francisco
Visa
Sponsors visas

Open. First seen 2 November 2026.

Apply on the company's site →

About the role

About the Role

Build evaluation systems for a web data extraction product, defining and measuring output quality across diverse websites, formats, and edge cases. You will own the process from evaluation methodology and datasets through to insights that inform model and product decisions.

What You'll Do

  • Design metrics that define high-quality structured extraction across varied web pages and data formats.
  • Build scalable pipelines and evaluation harnesses to measure output quality rigorously.
  • Generate and curate trustworthy datasets with reference extractions and ground-truth labels.
  • Connect evaluation results to model and product decisions, closing the feedback loop.
  • Make evaluation findings actionable for the broader team.

What We're Looking For

  • Four or more years of relevant experience, including at least two years working on evaluation systems.
  • Strong Python skills and experience with machine learning, LLMs, data pipelines, web scraping, NLP, evaluation frameworks, and statistical analysis.
  • Experience designing and building evaluations for LLM-based structured extraction from messy, real-world web pages or documents, including field-level accuracy measurement at scale.
  • Background building or evaluating large-scale web or document extraction systems and creating ground-truth datasets.
  • Statistical rigor in metric design and interpretation, including sampling, significance, estimator bias and variance, error bounds, and judge-to-human agreement.
  • Ability to combine research methodology with hands-on engineering and use evaluation results to guide real model or product changes.

Compensation & Benefits

Base salary range: $250,000 to $290,000 USD annually. Visa sponsorship is available.

Location

Hybrid, based in San Francisco, California. The role requires working on-site five days per week in San Francisco or Toronto.

What this role asks for

Particular to this role

NLP

Rare in True Talent Solutions Partners's other 128 open postings.

Standard for software engineering roles

Python

Common in software engineering postings at any company.

Compared with True Talent Solutions Partners's other open jobs.

The team: Client Roles

This role is one of 129 open in Client Roles. 129 were posted in the last 30 days, including 6 lead roles: the team is hiring the people who run it too.

WHAT THE TEAM WORKS WITH, ACROSS ITS POSTINGS

Python (70 of 129), LLM (43 of 129), TypeScript (42 of 129), React (37 of 129), AWS (29 of 129), Machine Learning (28 of 129), JavaScript (26 of 129)

OTHER ROLES ON THIS TEAM

From the team and dates on True Talent Solutions Partners's own job board.

Posting history

Reposted once since 30 October 2026: a posting for this role in San Francisco was taken down and the role was posted again.

Similar jobs

More jobs like this

SKILLSPython 46,696Machine Learning 19,872LLM 18,669NLP 2,447
PLACESSan Francisco 7,758United States 73,399
COMPANYJobs at True Talent Solutions Partners 129

Is this your posting? Email info@careerholo.com to have it removed.