True Talent Solutions Partners

Research Crawling Engineer

Remote (United States)
Salary
$160,000 - $250,000
Level
Mid
Work type
Remote · US only
Visa
Not offered

This posting has closed on 12 November 2026.

About the role

About the Role

Join a data infrastructure team building large-scale web data collection systems for AI research and model development. This role focuses on reliable, high-volume crawling and dataset preparation across diverse and changing web environments.

What You'll Do

  • Build and maintain web crawlers across social, travel, and multilingual websites.
  • Design high-throughput, fault-tolerant systems capable of collecting millions to billions of URLs per day.
  • Address anti-bot systems, rate limits, and dynamic websites using browser-based and other crawling techniques.
  • Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.
  • Collaborate with research teams to create datasets suited to model training and research needs.
  • Monitor crawl coverage, performance, and data quality, adapting systems as websites change.
  • Optimize crawling infrastructure for cost, latency, and reliability across cloud or bare-metal environments.

What We're Looking For

  • At least 2 years of experience building web crawlers or large-scale data pipelines; typically 4 or more years, with 2 or more years considered for candidates with a PhD.
  • Experience operating data pipelines at terabyte scale or larger, including high-volume crawling or data acquisition.
  • Strong programming skills in Go, Rust, Python, Java, or C++.
  • Deep knowledge of HTTP, networking, browser behavior, distributed systems, and parallel processing.
  • Hands-on experience navigating anti-bot systems, rate limits, and JavaScript-heavy websites.
  • Experience with headless browsers such as Playwright or Puppeteer, proxy systems, and IP rotation.
  • Background in data quality evaluation, web data processing, or cloud and bare-metal infrastructure.
  • Experience with NLP pipelines or machine learning dataset curation is a plus.

Compensation & Benefits

Annual salary range: $160,000 to $250,000 USD. Visa sponsorship is not available.

Location

Remote.

The team: Client Roles

This role is one of 128 open in Client Roles. 128 were posted in the last 30 days, including 7 lead roles: the team is hiring the people who run it too.

WHAT THE TEAM WORKS WITH, ACROSS ITS POSTINGS

Python (69 of 128), TypeScript (44 of 128), React (39 of 128), LLM (38 of 128), Machine Learning (34 of 128), JavaScript (28 of 128), AWS (26 of 128)

OTHER ROLES ON THIS TEAM

From the team and dates on True Talent Solutions Partners's own job board.

Posting history

Reposted 4 times since 27 October 2026: a posting for this role in Remote (United States) was taken down and the role was posted again.

Similar jobs

More jobs like this

Is this your posting? Email [email protected] to have it removed.