True Talent Solutions Partners

Research Engineer, Evals

San Francisco
Salary
$250,000 - $290,000 annually
Level
Mid
Work type
On-site · San Francisco
Visa
Sponsors visas

Open. First seen 1 November 2026.

Apply on the company's site →

About the role

About the Role

Build evaluation systems that measure the quality of large-scale web data extraction. You will define what good output means, develop the systems and datasets to assess it, and connect evaluation results to model and product decisions.

What You'll Do

  • Design rigorous metrics for extraction quality across diverse websites, formats, and edge cases.
  • Build scalable pipelines and evaluation harnesses to measure output quality.
  • Generate and curate trustworthy datasets with reference extractions and ground-truth labels.
  • Use evaluation results to guide model and product decisions, closing the feedback loop from measurement to improvement.
  • Make evaluation findings clear and actionable for the broader team.

What We're Looking For

  • At least 2 years of relevant evaluation experience, with 4 to 10 or more years of overall experience.
  • Experience designing and building evaluation systems for LLM-based structured extraction from messy, real-world web pages or documents, including field-level accuracy measurement at scale.
  • Experience building or evaluating large-scale web or document scraping and extraction systems, and creating ground-truth datasets.
  • Strong Python skills and experience with machine learning, LLMs, data pipelines, evaluation frameworks, web scraping, and NLP.
  • Statistical rigor in metric design and interpretation, including sampling, estimator bias and variance, significance, error bounds, and judge-to-human agreement.
  • A combination of research methodology and hands-on engineering, with examples of evaluations informing real model or product decisions.

Compensation & Benefits

Base salary range is $250,000 to $290,000 USD annually. Visa sponsorship is available.

Location

Hybrid, with a primary location in San Francisco, California. The role requires working on-site five days per week in San Francisco or Toronto, with relocation before starting if needed.

What this role asks for

Particular to this role

NLP

Rare in True Talent Solutions Partners's other 123 open postings.

Standard for software engineering roles

Python

Common in software engineering postings at any company.

Compared with True Talent Solutions Partners's other open jobs.

The team: Client Roles

This role is one of 124 open in Client Roles. 124 were posted in the last 30 days, including 6 lead roles: the team is hiring the people who run it too.

WHAT THE TEAM WORKS WITH, ACROSS ITS POSTINGS

Python (71 of 124), LLM (45 of 124), TypeScript (41 of 124), React (35 of 124), Machine Learning (32 of 124), AWS (31 of 124), JavaScript (24 of 124)

OTHER ROLES ON THIS TEAM

From the team and dates on True Talent Solutions Partners's own job board.

Other openings for this role

2 postings for this role in San Francisco since 30 October 2026, 1 still open. None has been taken down and posted again.

  • 1 November 2026 – open (this posting)
  • 30 October 2026 – closed 1 November 2026

Similar jobs

More jobs like this

SKILLSPython 46,655Machine Learning 19,863LLM 18,664NLP 2,449
PLACESSan Francisco 7,753United States 73,335
COMPANYJobs at True Talent Solutions Partners 124

Is this your posting? Email info@careerholo.com to have it removed.