True Talent Solutions Partners

Research Crawling Engineer

Remote (United States)
Salary
$160K - $250K
Level
Mid
Work type
Remote · US only
Visa
Not offered

Open. First seen 18 November 2026.

About the role

About the Role

Build large-scale web crawling and data acquisition systems that support AI research and model development. You will work closely with research teams to deliver high-quality web datasets for training and inference.

What You'll Do

  • Build and operate crawlers across diverse domains, including social media, travel, and multilingual websites.
  • Design high-throughput, fault-tolerant collection systems that process millions to billions of URLs per day.
  • Adapt crawling approaches to anti-bot systems, rate limits, and dynamic or JavaScript-heavy websites.
  • Develop large-scale pipelines for cleaning, deduplication, filtering, and normalization of web data.
  • Collaborate with research teams to shape datasets for research and model training.
  • Monitor crawl performance, coverage, and data quality, and respond to changes in web environments.
  • Optimize infrastructure for cost, latency, and reliability across cloud and bare-metal environments.

What We're Looking For

  • Typically 4 or more years of relevant experience building web crawlers or large-scale data acquisition systems. Candidates with a PhD may qualify with 2 or more years; hands-on crawler and pipeline experience is essential.
  • Experience operating distributed data pipelines at terabyte or larger scale, ideally processing millions to billions of URLs per day.
  • Strong proficiency in Go, Rust, Python, Java, or C++ and a solid understanding of distributed systems and parallel processing.
  • Deep knowledge of HTTP, networking, and browser behavior, including experience handling anti-bot systems, rate limits, and dynamic sites.
  • Experience with headless browsers such as Playwright or Puppeteer, Chrome DevTools Protocol, proxy systems, or IP rotation.
  • Experience with web data quality, benchmarking, cleaning, deduplication, and normalization; NLP pipelines or ML dataset curation are a plus.
  • Experience running crawling or data workloads on cloud platforms or bare-metal infrastructure. A degree in computer science, engineering, or a related field is a plus.

Compensation & Benefits

Base salary is $160,000 to $250,000 annually. Visa sponsorship is not available.

Location

Remote.

What this role asks for

Standard for software engineering roles

Python

Common in software engineering postings at any company.

Compared with True Talent Solutions Partners's other open jobs.

The team: Client Roles

This role is one of 128 open in Client Roles. 128 were posted in the last 30 days, including 7 lead roles: the team is hiring the people who run it too.

WHAT THE TEAM WORKS WITH, ACROSS ITS POSTINGS

Python (69 of 128), TypeScript (44 of 128), React (39 of 128), LLM (38 of 128), Machine Learning (34 of 128), JavaScript (28 of 128), AWS (26 of 128)

OTHER ROLES ON THIS TEAM

From the team and dates on True Talent Solutions Partners's own job board.

Posting history

Reposted 4 times since 27 October 2026: a posting for this role in Remote (United States) was taken down and the role was posted again.

Similar jobs

More jobs like this

Is this your posting? Email [email protected] to have it removed.