Backend Engineer (Data Acquisition)

• You’ll join our Crawler Engineering team as a Backend Engineer focused on data acquisition
• This is a building role: you’ll design and develop high-performance systems, mostly in Rust, that collect billions of data points from the web
• The work sits where backend engineering meets how the web works: HTTP stacks, proxies and browser tooling
• When a data source blocks an approach, you’ll dig into what’s happening on the wire and find another way forward
• You’ll also help make our infrastructure “agent-friendly”, so other teams can safely build on it with AI
• Design, build and improve backend systems for data acquisition that handle close to 200 million requests a day
• Work on our in-house proxy layer and browser tooling: request routing, connection handling and protocol-level behaviour
• Investigate hard crawler problems, test alternative approaches when one gets blocked, and turn what works into robust, reusable systems
• Build clear interfaces and guardrails so other teams and AI agents can use what you build reliably
• Own the quality of your code, including code written with AI assistance: tests, observability and maintainability
• Work closely with crawler engineers, DevOps and Security to ship solutions that are secure and scalable
• The Crawler engineering team makes sure Lighthouse customers get fresh, high-quality rates, ranking and supply data. We handle close to 200 million requests a day across 200+ data sources, so the team carries a big responsibility for keeping our crawling infrastructure reliable, efficient and ahead of a web that keeps changing
• —-Technologies you will work with—-
• Rust, TypeScript/Node.js, Python, Google Cloud, Kubernetes, Prometheus, Grafana, Claude Code

Benefits

• Vision & dental Insurance
• Work from home opportunities
• Health insurance

• You have a solid grasp of how the web works under the hood: HTTP, TCP/IP, TLS and proxies
• You have strong analytical and debugging skills. You’re comfortable with packet captures, profilers and reading other people’s source code
• You have an R&D mindset. You’re curious about how browsers and network stacks work internally, you’re comfortable with ambiguity, and you’d rather prove a hypothesis than assume it
• You’re fluent in English
• You’re experienced with a systems language such as Rust or C/C++, or you have a strong track record in another backend language and a real drive to go deep in Rust
• You have hands on experience building, deploying and running production backend services in a cloud environment
• You communicate clearly, share your opinions, welcome feedback and work well in a team
• Not ticking every box? No problem! We value diverse backgrounds and unique skill sets, and we encourage individuals from all walks of life to apply. If your experience looks a little different from what we’ve described, but you’re passionate about what we do and are a quick learner, we’d love to hear from you!