Senior Software Engineer (LLM Ops & Evals)
Posted Updated
- Own the LLM gateway: routing, provider failover, rate limits, retries, and cost attribution across multiple model providers
- Deploy, version and deprecate models across clouds, regions and environments, including quota and capacity planning, managed as infrastructure as code
- Build the shared evaluation platform: versioned datasets, experiment tracking, run and result schemas, reporting, and trace linkage back to the run
- Own the infrastructure for async and long-running AI workloads
- Own observability for AI traffic: latency, retries and fallbacks, token usage, cost and errors, per team and per use case
- Take part in the on-call rotation, run incidents, and close the follow-ups
- Implement the security and compliance controls the platform is held to: retention, access control, RBAC and SSO
- Define and maintain clean integration contracts between the platform and the teams that consume it
- Partner with product and ML engineers to turn their requirements into platform capabilities that are self-service rather than a request queue
- Comfort with privacy and compliance work: PII handling, anonymisation, retention, access control
- Experience with cloud at the infrastructure level and infrastructure as code (we run across Azure and GCP with Terraform)
- Experience building platform or shared-service capabilities consumed by multiple internal teams
- 5+ years in backend or platform engineering, with strong production Python
- Experience running a shared service in production: on-call, incidents, postmortems, SLOs
- Experience building or running LLM inference infrastructure: a gateway or routing layer with multiple providers, failover, rate limiting and cost attribution
- Hands-on experience with observability tools (OpenTelemetry, Grafana), including instrumenting services and designing dashboards and alerts
- Judgment: You exercise sound judgment in ambiguous situations, balance speed and accuracy, and adjust priorities proactively
- Ownership: You own work end-to-end, anticipate issues, and ensure high-quality delivery without close supervision
- Adaptability: You navigate ambiguity calmly, model positive behavior, and help peers adjust through clear communication
- Collaboration: You build strong cross-functional relationships and influence peers through expertise, data, and empathy
- Growth Mindset: You encourage open feedback exchange and provide clear, balanced feedback that helps others grow
- Synthetic data or document anonymisation pipelines
- Document AI: VLMs, OCR, structured extraction and the metrics that go with it
- Domain experience in audit, accounting or fintech
- Temporal or another durable workflow engine
- Self-hosted inference, capacity planning, load testing
- Experience with LLM or agent evaluation
- Recruiter screen
- Hiring Manager interview
- Peer programming session
- System design interview
- Final interviews with Engineering leadership