← All jobs
True Talent Solutions Partners

Research Crawling Engineer

Remote (United States)
Salary
$160,000 - $250,000 annually
Level
Mid
Work type
Remote · US only
Visa
Not offered

Open. First seen 25 October 2026.

Apply on the company's site →

About the role

About the Role

Join a data infrastructure team building large-scale web crawling systems that supply public web data for AI research and model development. You will work closely with research teams to shape data collection around their modeling needs.

What You'll Do

  • Build and maintain web crawlers across diverse domains, including social, travel, and multilingual websites.
  • Design high-throughput, fault-tolerant systems that collect millions to billions of URLs per day.
  • Adapt collection methods to anti-bot protections, rate limits, and dynamic, JavaScript-heavy sites.
  • Build pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.
  • Develop and maintain datasets for research and model training in collaboration with research teams.
  • Monitor crawl coverage, performance, and data quality, and respond to changing web environments.
  • Optimize crawling infrastructure for cost, latency, and reliability across cloud and bare-metal environments.

What We're Looking For

  • At least 2 years of experience building web crawlers or large-scale data pipelines; typically 4 or more years, with 2 or more years considered for candidates with a PhD.
  • Experience operating data pipelines at terabyte scale and building distributed or parallel systems.
  • Strong programming skills in Go, Rust, Python, Java, or C++.
  • Deep knowledge of HTTP, networking, and browser behavior, with experience handling anti-bot systems, rate limits, or JavaScript-heavy sites.
  • Experience with headless browsers such as Playwright or Puppeteer, proxy systems, and IP rotation.
  • Background in data quality evaluation, web data processing, NLP pipelines, or ML dataset curation is beneficial.
  • Experience running data workloads on cloud platforms or bare-metal infrastructure.

Compensation & Benefits

Base salary: $160,000 to $250,000 USD annually. Visa sponsorship is not available.

Location

Remote.

What this role asks for

Standard for software engineering roles

Python

Common in software engineering postings at any company.

Compared with True Talent Solutions Partners's other open jobs.

The team: Client Roles

This role is one of 121 open in Client Roles. 121 were posted in the last 30 days, including 6 lead roles: the team is hiring the people who run it too.

WHAT THE TEAM WORKS WITH, ACROSS ITS POSTINGS

Python (65 of 121), LLM (48 of 121), TypeScript (47 of 121), React (38 of 121), Machine Learning (31 of 121), AWS (27 of 121), JavaScript (27 of 121)

OTHER ROLES ON THIS TEAM

From the team and dates on True Talent Solutions Partners's own job board.

Similar jobs

Is this your posting? Email info@careerholo.com to have it removed.