True Talent Solutions Partners

Research Engineer, Evals

San Francisco
Salary
$250,000 - $290,000 annually
Level
Mid
Work type
On-site · San Francisco
Visa
Sponsors visas

Open. First seen 6 November 2026.

Apply on the company's site →

About the role

About the Role

Build evaluation systems for a B2B SaaS product that turns web content into structured data for AI applications. You will define how output quality is measured, build the systems that evaluate it, and connect results to model and product decisions.

What You'll Do

  • Design metrics for evaluating output quality across diverse websites, formats, and edge cases.
  • Build evaluation pipelines and harnesses that measure quality rigorously and at scale.
  • Generate and curate datasets that support trustworthy evaluation.
  • Connect evaluation results to model and product decisions, helping teams understand where systems succeed and where they need improvement.

What We're Looking For

  • Typically 4 to 10 or more years of relevant experience, including at least 2 years working on evaluation systems.
  • Strong Python skills and experience with machine learning, LLMs, data pipelines, web scraping, NLP, evaluation frameworks, and statistical analysis.
  • Experience building evaluations for structured extraction from messy, real-world web pages or documents, including field-level accuracy measurement at scale.
  • Ability to design evaluation metrics and systems, build ground-truth datasets, and apply statistical rigor, including sampling, error bounds, and judge-to-human agreement.
  • Experience combining research methodology with hands-on engineering and using evaluation findings to inform model or product decisions.

Compensation & Benefits

Base salary is $250,000 to $290,000 USD annually. Visa sponsorship is available.

Location

Hybrid, with the primary location in San Francisco, California. The role is also available in Toronto, Canada, with five days per week on-site.

What this role asks for

Particular to this role

NLP

Rare in True Talent Solutions Partners's other 125 open postings.

Standard for software engineering roles

Python

Common in software engineering postings at any company.

Compared with True Talent Solutions Partners's other open jobs.

The team: Client Roles

This role is one of 125 open in Client Roles. 125 were posted in the last 30 days, including 6 lead roles: the team is hiring the people who run it too.

WHAT THE TEAM WORKS WITH, ACROSS ITS POSTINGS

Python (71 of 125), TypeScript (47 of 125), LLM (45 of 125), React (42 of 125), Machine Learning (36 of 125), AWS (31 of 125), JavaScript (27 of 125)

OTHER ROLES ON THIS TEAM

From the team and dates on True Talent Solutions Partners's own job board.

Posting history

Reposted 5 times since 30 October 2026: a posting for this role in San Francisco was taken down and the role was posted again.

Similar jobs

More jobs like this

SKILLSPython 47,633Machine Learning 20,147LLM 18,882NLP 2,457
PLACESSan Francisco 7,719United States 74,714
COMPANYJobs at True Talent Solutions Partners 125

Is this your posting? Email info@careerholo.com to have it removed.