Member of Technical Staff (Web Access Engineer, Crawler)
- Level
- Staff
- Work type
- Hybrid · London
Not stated in the posting: salary, visa.
Open. First seen 13 November 2026.
About the role
Perplexity is hiring a Web Access Engineer for the Crawler team to build reliable fetching and rendering systems for AI-powered search. You'll work on freely available public web content, within site permissions and approved access policies, and develop the infrastructure that supplies content to downstream indexing and retrieval systems.
Your focus will be diagnosing network errors, page-loading issues, and incomplete rendering. You'll trace problems from network protocols through client-side execution, distinguish technical failures from intentional access restrictions, and turn findings into production fixes that improve content quality and efficiency at scale.
Responsibilities
- Own fetching and rendering improvements end to end, from investigation through implementation and production rollout, partnering with crawler engineers and SREs.
- Investigate fetch and rendering failures across DNS, TLS, HTTP, caching, redirects, and client-side rendering, and classify root causes.
- Improve page loading, completion detection, and content capture to preserve relevant content and handle dynamic pages reliably.
- Ensure the crawler respects site-imposed access restrictions and rate limits, identifies itself clearly, and retries only transient technical errors.
- Build tools to reproduce failures, compare crawler output against expected page content, and turn findings into regression tests.
- Define metrics for fetching reliability, content completeness, and error classification, and use them to prioritize improvements.
Qualifications
- 5+ years of software engineering experience in web infrastructure, browsers, CDNs, or crawling, with strong programming skills in at least one of Go, Python, Rust, or C++.
- Deep understanding of HTTP/1.1, HTTP/2, and HTTP/3, including status semantics, caching, conditional requests, content encoding, and redirects; strong knowledge of TLS, DNS, cookies, and sessions.
- Understanding of CDN caching and content delivery, HTTP error handling, retry backoff, and server-defined rate limits.
- Knowledge of page lifecycle, network idle, service workers, client-side rendering, and shadow DOM; experience debugging JavaScript and fluency with Chrome DevTools, CDP, HAR files, curl, and packet captures.
- Comfort working independently when the path forward is not fully defined.
Nice to have
- Experience developing CDN, caching, or web performance systems.
- Experience operating crawlers, headless browsers, or distributed content-acquisition systems at scale.
- Contributions to a browser engine or JavaScript runtime.
What this role asks for
Particular to this role
C++ · JavaScript · Networking
Rare in Perplexity's other 95 open postings.
Standard for software engineering roles
Python
Common in software engineering postings at any company.
Compared with Perplexity's other open jobs.
The team: Search HIRES STEADILY
This role is one of 17 open in Search. 4 were posted in the last 30 days and the oldest in March: a team that hires all year rather than one being built.
WHAT THE TEAM WORKS WITH, ACROSS ITS POSTINGS
Rust (12 of 17), C++ (9 of 17), Python (9 of 17), Golang (8 of 17), Kubernetes (7 of 17), LLM (7 of 17), AWS (6 of 17)
OTHER ROLES ON THIS TEAM
- Member of Technical Staff (Index Engineer, Search Crawler) posted Nov 17
- Member of Technical Staff (Site Reliability Engineer, Crawler) posted Nov 13
- Member of Technical Staff (Software Engineer, Fraud Prevention) posted Oct 23
- Member of Technical Staff (Search Crawler Analyst) posted Oct 8
- Member of Technical Staff (Machine Learning Engineer, Search Agent RL) posted Oct 6
- Member of Technical Staff (Search Core DevOps Engineer) posted Sep 25
- Member of Technical Staff (Search Core, Indexing & Data Pipeline Stream) posted Sep 18
- Member of Technical Staff (Search Quality Analyst) posted Sep 2
- Member of Technical Staff (Machine Learning Engineer, Ranking Quality - Search) posted Aug 12
- Internship - Search Backend Infra Engineer posted Jul 1
- Internship - Search Machine Learning Engineer posted Apr 8
- Internship - Search Machine Learning Engineer posted Jan 13
- Member of Technical Staff (Backend Software Engineer, Search Crawler) posted Jul 28
- Member of Technical Staff (Search Engine Developer, Search Core) posted Mar 31
- Member of Technical Staff (Machine Learning Engineer, Search) posted Mar 25
- Member of Technical Staff (Backend/Infrastructure Engineer, Search) posted Mar 7
From the team and dates on Perplexity's own job board.
Similar jobs
- Staff Software Engineer, Core Platform at Judi Health
Shares 5 skills: Networking, Rust, Golang… · Staff level - Senior Technical Support Engineer, APJ at Horizon3
Shares 4 skills: Networking, Golang, JavaScript… - Senior Customer Engineer, Digital Native Start ups at Cloudflare
Shares 4 skills: Networking, Golang, JavaScript… - Senior Software Engineer, Platform Networking at Crusoe
Shares 5 skills: Networking, Rust, Golang… - Principal Engineer, Network Software (Traffic & Modeling) at Vultr
Shares 4 skills: Networking, Rust, Golang… · Staff level - Member of Technical Staff (Index Engineer, Search Crawler) at Perplexity
Also at this company
More jobs like this
Is this your posting? Email [email protected] to have it removed.