True Talent Solutions Partners

Research Crawling Engineer

Remote (United States)
Salary
$160K - $250K
Level
Mid
Work type
Remote · US only
Visa
Not offered

This posting has closed on 15 November 2026.

About the role

About the Role

Join a data and infrastructure team building large-scale web data systems for AI research and model development. You will help deliver reliable, high-quality datasets by operating crawlers across complex and changing web environments.

What You'll Do

  • Build and maintain large-scale crawlers across diverse domains, including social media, travel, and multilingual websites.
  • Design high-throughput, fault-tolerant collection systems capable of processing millions to billions of URLs per day.
  • Adapt crawling strategies to anti-bot measures, rate limits, and dynamic or JavaScript-heavy sites.
  • Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.
  • Build datasets for research and model training in collaboration with research teams.
  • Monitor crawl coverage, performance, and data quality, and improve systems as web environments change.
  • Optimize infrastructure for cost, latency, and reliability across cloud and bare-metal environments.

What We're Looking For

  • Typically 4 or more years of relevant experience building web crawlers or large-scale data acquisition systems. Candidates with a PhD may qualify with 2 or more years.
  • Strong experience with Go, Rust, Python, Java, or C++, and distributed systems or parallel processing.
  • Experience operating data pipelines at terabyte scale and processing millions to billions of URLs per day.
  • Deep understanding of HTTP, networking, and browser behavior, including experience handling anti-bot systems, rate limits, and JavaScript-heavy websites.
  • Experience with headless browsers such as Playwright or Puppeteer, proxy systems, IP rotation, and request orchestration is valuable.
  • Background in data quality evaluation, web data processing, NLP pipelines, or machine-learning dataset curation is a plus.
  • Experience running data workloads on cloud or bare-metal infrastructure; a relevant technical degree is beneficial.

Compensation & Benefits

Base salary: $160,000 to $250,000 USD annually. Visa sponsorship is not available.

Location

Remote.

The team: Client Roles

This role is one of 128 open in Client Roles. 128 were posted in the last 30 days, including 7 lead roles: the team is hiring the people who run it too.

WHAT THE TEAM WORKS WITH, ACROSS ITS POSTINGS

Python (69 of 128), TypeScript (44 of 128), React (39 of 128), LLM (38 of 128), Machine Learning (34 of 128), JavaScript (28 of 128), AWS (26 of 128)

OTHER ROLES ON THIS TEAM

From the team and dates on True Talent Solutions Partners's own job board.

Posting history

Reposted 4 times since 27 October 2026: a posting for this role in Remote (United States) was taken down and the role was posted again.

Similar jobs

More jobs like this

Is this your posting? Email [email protected] to have it removed.