Research Crawling Engineer
- Salary
- $160,000 - $250,000
- Level
- Mid
- Work type
- Remote · US only
- Visa
- Not offered
This posting has closed on 12 November 2026.
About the role
About the Role
Join a data infrastructure team building large-scale web data collection systems for AI research and model development. This role focuses on reliable, high-volume crawling and dataset preparation across diverse and changing web environments.
What You'll Do
- Build and maintain web crawlers across social, travel, and multilingual websites.
- Design high-throughput, fault-tolerant systems capable of collecting millions to billions of URLs per day.
- Address anti-bot systems, rate limits, and dynamic websites using browser-based and other crawling techniques.
- Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.
- Collaborate with research teams to create datasets suited to model training and research needs.
- Monitor crawl coverage, performance, and data quality, adapting systems as websites change.
- Optimize crawling infrastructure for cost, latency, and reliability across cloud or bare-metal environments.
What We're Looking For
- At least 2 years of experience building web crawlers or large-scale data pipelines; typically 4 or more years, with 2 or more years considered for candidates with a PhD.
- Experience operating data pipelines at terabyte scale or larger, including high-volume crawling or data acquisition.
- Strong programming skills in Go, Rust, Python, Java, or C++.
- Deep knowledge of HTTP, networking, browser behavior, distributed systems, and parallel processing.
- Hands-on experience navigating anti-bot systems, rate limits, and JavaScript-heavy websites.
- Experience with headless browsers such as Playwright or Puppeteer, proxy systems, and IP rotation.
- Background in data quality evaluation, web data processing, or cloud and bare-metal infrastructure.
- Experience with NLP pipelines or machine learning dataset curation is a plus.
Compensation & Benefits
Annual salary range: $160,000 to $250,000 USD. Visa sponsorship is not available.
Location
Remote.
The team: Client Roles
This role is one of 128 open in Client Roles. 128 were posted in the last 30 days, including 7 lead roles: the team is hiring the people who run it too.
WHAT THE TEAM WORKS WITH, ACROSS ITS POSTINGS
Python (69 of 128), TypeScript (44 of 128), React (39 of 128), LLM (38 of 128), Machine Learning (34 of 128), JavaScript (28 of 128), AWS (26 of 128)
OTHER ROLES ON THIS TEAM
- Senior Engineering Manager posted Nov 18
- Senior AI/ML Engineer posted Nov 18
- Senior Software Engineer posted Nov 18
- Software Engineer, Distributed Systems (Core) posted Nov 18
- Product Manager, Supply Chain posted Nov 18
- Product Lead, Agentic Personalization posted Nov 18
- Senior Software Engineer, Autonomous Systems posted Nov 18
- Full-Stack Software Engineer posted Nov 18
- Senior AI Scheduler posted Nov 18
- Product Manager, AI Platform posted Nov 18
- Head of Software Architecture, Embodied AI Platform posted Nov 18
- Senior/Staff Software Engineer posted Nov 18
- Agent Product Engineer posted Nov 18
- Product Engineer posted Nov 18
- Agent Deployment Engineer posted Nov 18
- Machine Learning Engineer posted Nov 18
- Forward Deployed Engineer posted Nov 18
- Founding Research Engineer, Robot Learning posted Nov 18
- Founding Engineer posted Nov 18
- Research Crawling Engineer posted Nov 18
- Founding Product Engineer posted Nov 18
- Head of RF Engineering posted Nov 18
- Software Engineer, Security posted Nov 18
- Senior Software Engineer posted Nov 18
- Senior Forward Deployed Engineer posted Nov 18
- Senior Software Engineer posted Nov 18
- Founding Engineer, Applied AI posted Nov 18
- Founding Product Designer posted Nov 18
- Member of Technical Staff, Applied ML posted Nov 18
- Founding AI Agent Engineer posted Nov 18
- Senior Software Engineer, Customer Studio Backend posted Nov 18
- Founding ML Research Engineer posted Nov 18
- Senior Forward Deployed Engineer, Operate posted Nov 18
- Founding Product Engineer posted Nov 18
- Senior/Staff Software Engineer, Platform posted Nov 18
- Senior/Staff Full Stack Engineer posted Nov 18
- Software Engineer, C++ Systems posted Nov 18
- Founding Software Engineer posted Nov 18
- Engineering Manager, Security Platform posted Nov 18
- Senior Software Engineer, Design posted Nov 18
- Lead Platform Engineer posted Nov 18
- Technical Product Manager, Senior posted Nov 18
- Member of Technical Staff, Full Stack posted Nov 18
- Software Engineer, AI Agents posted Nov 18
- Staff Product Manager, AI posted Nov 18
- Product Engineer posted Nov 18
- Fullstack Software Engineer posted Nov 18
- Founding AI Engineer posted Nov 18
- Software Engineer, iOS posted Nov 18
- Senior Infrastructure Engineer posted Nov 18
- Principal Software Engineer, Perception posted Nov 18
- Full Stack Engineer posted Nov 18
- Member of Technical Staff, Reinforcement Learning posted Nov 18
- Solutions Engineer posted Nov 18
- Electrical & Computer Engineer posted Nov 18
- Senior Power Electronics Engineer posted Nov 18
- Lead Product Engineer posted Nov 18
- Senior Solutions Engineer, Digital Native and Mid-Market posted Nov 18
- Mid/Senior Full Stack AI Engineer posted Nov 18
- Staff Product Manager posted Nov 18
- Engineering Lead posted Nov 18
- Forward Deployed Product Manager posted Nov 18
- Director of Engineering (Product) posted Nov 18
- Founding Product Designer posted Nov 18
- Founding Senior Engineer posted Nov 18
- Staff Full-Stack Engineer, Core Application posted Nov 18
- Robotics Software Engineer posted Nov 18
- Technical Recruiter posted Nov 18
- Audiobook QA Expert, Dutch posted Nov 18
- Senior Applied AI Product Designer posted Nov 18
- Member of Technical Staff, ML Systems posted Nov 18
- Principal Analog/Mixed Signal Engineer posted Nov 18
- Lead Software Engineer, Platform posted Nov 18
- Software Engineer, Marketplace posted Nov 18
- Lead AI Engineer posted Nov 18
- Electrical Engineer posted Nov 18
- Member of Technical Staff posted Nov 18
- Product Engineer, Mid-Senior posted Nov 17
- Founding Backend Software Engineer posted Nov 17
- Staff Software Engineer, Infrastructure posted Nov 17
- Senior Product Manager, Staffing and AI Agents posted Nov 17
- Founding Product Engineer posted Nov 17
- Software Engineer posted Nov 17
- Founding Full-Stack Engineer posted Nov 17
- Founding Backend & Infrastructure Engineer posted Nov 17
- Senior Software Engineer, Backend posted Nov 17
- Software Engineer, Agentic Systems posted Nov 17
- Senior/Staff Product Engineer, Applied AI posted Nov 17
- Founding Forward Deployed Engineer posted Nov 17
- Senior / Staff Product Engineer posted Nov 17
- Senior Technical Product Manager, Data Integrations posted Nov 17
- Member of Technical Staff, Agent Platform posted Nov 17
- Senior Analog/Mixed-Signal Engineer posted Nov 17
- Founding Forward Deployed Engineer posted Nov 17
- Data Architect (m/w/d) posted Nov 17
- Mechanical Design Engineer posted Nov 17
- Senior Full-Stack Software Engineer posted Nov 17
- Founding Member of Technical Staff posted Nov 16
- Engineering Manager posted Nov 16
- Senior Data & Machine Learning Engineer posted Nov 16
- Forward Deployed Software Engineer, Tactical Edge posted Nov 16
- Founding RL Engineer posted Nov 16
- Senior Quantitative Developer posted Nov 16
- Founding Forward Deployed Engineer posted Nov 16
- Founding Forward Deployed Engineer posted Nov 16
- Senior Security Engineer posted Nov 16
- Senior Platform Engineer posted Nov 16
- Member of Technical Staff, Research Engineering posted Nov 16
- Forward Deployed Infrastructure Engineer posted Nov 16
- Senior Control Electronics Engineer posted Nov 15
- Robotics Infrastructure Engineer posted Nov 15
- Full Stack Engineer posted Nov 15
- Founding Engineer posted Nov 15
- GTM Engineer posted Nov 15
- Full-Stack Engineer, Agent Evaluation & Reliability posted Nov 15
- Founding Fullstack Engineer posted Nov 15
- Founding Engineer posted Nov 15
- Mechanical Engineering Expert (MSc or PhD) posted Nov 15
- Mechanical Engineer posted Nov 15
- Computer Vision ML / Research Engineer posted Nov 15
- Forward Deployed Engineer posted Nov 15
- Member of Technical Staff posted Nov 15
- Technical Recruiter posted Nov 15
- Product Manager, AI Education posted Nov 15
- Agent Deployment Engineer posted Nov 15
- Senior Brand & Identity / Presentation Designer posted Nov 15
- Software Engineer posted Nov 15
- Senior Full-Stack Engineer posted Nov 12
From the team and dates on True Talent Solutions Partners's own job board.
Posting history
Reposted 4 times since 27 October 2026: a posting for this role in Remote (United States) was taken down and the role was posted again.
- 18 November 2026 – open
- 12 November 2026 – closed 15 November 2026
- 8 November 2026 – closed 12 November 2026 (this posting)
- 2 November 2026 – closed 6 November 2026
- 27 October 2026 – closed 31 October 2026
Similar jobs
- Research Crawling Engineer at Wynd Labs
Shares 9 skills: Playwright, NLP, Rust… · same seniority · same remote region - Dev AI Software Engineer Graduate (Dev Infra US) - 2027 Start at ByteDance
Shares 7 skills: NLP, Golang, Java… · same country - Machine Learning Engineer III at Peloton
Shares 6 skills: NLP, Golang, Java… · same country - QA Engineer at CUBE
Shares 6 skills: Playwright, NLP, Java… · same seniority - Senior Software Quality Assurance (SQA) Tester at LTS
Shares 5 skills: Playwright, Java, JavaScript… · same remote region - Senior AI/ML Engineer at True Talent Solutions Partners
Also at this company
More jobs like this
Is this your posting? Email [email protected] to have it removed.