Senior Backend Engineer (Inference Platform)

• Every model call from our products and customers passes through one layer: our inference gateway. It allows users to select a model and we intelligently route the call to optimize on cost , latency and high availability. We are now building the control plane above it, to manage and govern access, performance and cost
• You will own and scale that gateway. The work sits between model serving and product: routing and fallback across hosted providers and our own models, cache-aware routing decisions, metering accurate enough to bill on, and enforcement on the request path within a single-digit-millisecond budget
• Own the inference gateway that every model call routes through, including provider and model selection, fallback, and caps
• Build intelligence layer to autonomously pick the right model for the right turn in the task
• Make routing decisions that account for prefix-cache behavior, batching, and time to first token, not only price and availability
• Build cost attribution and metering across providers, including cached-token accounting, accurate enough to bill and enforce on
• Build request-path enforcement that degrades gracefully and never takes down a customer’s inference call
• Ship inference-time changes in shadow mode first, with the measurement to prove they work

Benefits

• A remote-first work environment with hubs in California, NYC, and Canada that offer monthly in-person gatherings
• Unlimited PTO with 11 U.S. holidays observed and a week shutdown in December to rest and recharge
• $500 work-from-home stipend to be used up to a year of your start date
• In-person coworking weeks 1 to 2 times a year
• Competitive health insurance plan, where 100% of the policyholder is covered and 75% for dependents
• 401k program, 3% match coming January 2025
• $1,200 / year Health & Wellness Allowance to support your personal goals
• Chance to collaborate with a team at the forefront of AI research
• 12 weeks of paid paternity leave in the US; additional time off is also considered

• Experience building or operating a proxy, gateway, or data plane on the critical path of production traffic, with sound instincts for failing open
• A strong backend generalist (Python, Kubernetes). We expect you to pick up whatever you have not touched yet
• Hands-on production experience with model serving on stacks such as vLLM, SGLang (or similar platforms), with a working understanding of continuous batching, KV and prefix caching, and the tradeoff between time to first token and throughput
• Fluency across provider APIs: tokenization, prompt-caching semantics, and the differences in how usage and cached tokens are reported and priced
• You have tuned a serving stack against live traffic and know what moves p99 in practice, not only in theory
• We weigh trajectory and recent growth more heavily than years of experience
• Comfort with streaming responses and the parts that get awkward: partial state, mid-stream failures, and work that can only happen once the stream closes
• Usage metering, billing, or cost optimization at scale
• Experience at an inference provider or GPU cloud, or contributions to an open-source serving stack
• Fine-tuning and evaluating models in production