True Talent Solutions Partners

Research Crawling Engineer

Remote (United States)
Salary
$160,000 - $250,000 annually
Level
Mid
Work type
Remote · US only
Visa
Not offered

Open. First seen 2 November 2026.

Apply on the company's site →

About the role

About the Role

Join a growing data infrastructure team building large-scale web data systems for AI research and model development. You will develop reliable crawling and data acquisition infrastructure across diverse websites, helping research teams create datasets for training and evaluating advanced models.

What You'll Do

  • Build and maintain web crawlers across social, travel, multilingual, and other domains.
  • Design high-throughput, fault-tolerant collection systems that process millions to billions of URLs per day.
  • Adapt crawling methods to anti-bot systems, rate limits, and dynamic or JavaScript-heavy websites.
  • Build pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.
  • Work with research teams to align data collection and dataset curation with modeling needs.
  • Monitor crawl coverage, performance, and data quality, and improve systems as websites change.
  • Optimize infrastructure for cost, latency, and reliability across cloud and bare-metal environments.

What We're Looking For

  • At least 2 years of experience building web crawlers or large-scale data pipelines; 4 or more years is preferred, with flexibility for candidates holding a PhD.
  • Experience operating data pipelines at terabyte scale or greater, including distributed systems or parallel processing.
  • Strong programming skills in Go, Rust, Python, Java, or C++.
  • Deep knowledge of HTTP, networking, browser behavior, and techniques for handling anti-bot systems, rate limits, and dynamic websites.
  • Familiarity with headless browsers such as Playwright or Puppeteer, proxy systems, and request orchestration.
  • Experience with large-scale data quality evaluation, NLP pipelines, or machine learning dataset curation is a plus.
  • Experience running data workloads on cloud platforms or bare-metal infrastructure.

Compensation & Benefits

Base salary is $160,000 to $250,000 annually. Visa sponsorship is not available.

Location

Fully remote.

What this role asks for

Standard for software engineering roles

Python

Common in software engineering postings at any company.

Compared with True Talent Solutions Partners's other open jobs.

The team: Client Roles

This role is one of 125 open in Client Roles. 125 were posted in the last 30 days, including 7 lead roles: the team is hiring the people who run it too.

WHAT THE TEAM WORKS WITH, ACROSS ITS POSTINGS

Python (64 of 125), TypeScript (45 of 125), LLM (44 of 125), React (41 of 125), JavaScript (28 of 125), AWS (24 of 125), Machine Learning (24 of 125)

OTHER ROLES ON THIS TEAM

From the team and dates on True Talent Solutions Partners's own job board.

Posting history

Reposted once since 27 October 2026: a posting for this role in Remote (United States) was taken down and the role was posted again.

  • 2 November 2026 – open (this posting)
  • 27 October 2026 – closed 31 October 2026

Similar jobs

More jobs like this

SKILLSPython 46,770Machine Learning 19,834JavaScript 15,407Java 14,134
PLACESUnited States 73,414
COMPANYJobs at True Talent Solutions Partners 125

Is this your posting? Email info@careerholo.com to have it removed.