Senior Software Engineer (LLM Platform)

• Handshake is hiring a Senior LLM Platform Engineer to join our Data and ML Platform team
• This team supports Handshake’s core career marketplace and Handshake AI (HAI) by building the shared data, ML, and LLM infrastructure behind production workflows
• This infrastructure-heavy role primarily owns our shared LLM control plane: LiteLLM gateways, provider integrations, shared clients, access controls, observability, cost attribution, capacity management, and hosted or self-hosted inference
• You’ll also contribute to adjacent platform systems for workflow orchestration, model serving, shared cloud infrastructure, and developer enablement—working closely with Backend Platform, HAI engineering, data science, and FDEs
• Build and operate our LiteLLM-based AI gateways and shared LLM clients
• Own provider and model onboarding, routing, failover, rate limits, and capacity planning
• Build self-service virtual-key, model-access, budget, and credential-management workflows
• Establish SLOs, observability, cost attribution, and alerts for production LLM traffic
• Safely qualify and roll out new models, providers, SDKs, and gateway configurations
• Support hosted and self-hosted inference through a consistent platform interface
• Partner with product, AI, and FDE teams to turn recurring delivery problems into paved-platform capabilities
• Contribute to the broader Data and ML Platform, including workflow orchestration, model serving, shared cloud infrastructure, and developer tooling
• Participate in team on-call and support, improving runbooks, automation, and reliability across owned platform services
• Our Stack:
• Python, TypeScript, Go, LiteLLM, FastAPI, PostgreSQL, Redis, GCP, Kubernetes, Terraform, Spacelift, BigQuery, Airflow, Dataflow/Beam, Datastream, OpenAI, Anthropic, Gemini, OpenRouter, vLLM, Modal, Anyscale/Ray, Datadog, Arize, Temporal, Cloudflare, and Tailscale

Benefits

• Health Insurance: Comprehensive medical , dental, and vision coverage
• Fertility and family forming: Through a partnership, Handshake offers comprehensive family benefits, including infertility treatments and a $15K stipend, to its employees.
• LGTBQ+ coverage: All medical plans cover transgender services.
• Mental health: Discounts on a variety of wellness apps and an Employee Assistance Program with access to five free counseling sessions.
• Tax savings: Tax-advantaged benefits to save money on healthcare, dependent care, and other expenses.
• Life & disability insurance: Essential benefits and automatic coverage with basic life insurance, plus short-term and long-term disability coverage.
• 401k match: Dollar-for-dollar match up to 1% of pay, with a max of $1,200 per year.
• Equity: Our equity program includes grants at hire, refresher grants after two years of tenure, and grants alongside promotions.
• Financial management: Handshake partners to provide free professional financial planning services.
• Learning & development: Employees are provided with an annual stipend to use towards professional development, career development workshops, manager trainings, and 1-on-1 coaching through external partnerships
• Community leader support: Employee Resource Group leaders are compensated with a quarterly bonus for the high-impact work the role requires
• Parental leave: Leaves of 16 weeks for birth-giving parents and 10 weeks for non-birth-giving parents; the policy applies to adoptions as well. Handshake also partners to provide one-on-one coaching on preparing for leave and coming back from leave
• Flexible time off: No accrued time off or waiting period to take vacation. Two paid flexible holiday days off to observe holidays that closely align with religious, cultural, and personal needs
• Sabbatical: Paid six-week sabbatical to recharge and reset beginning at six years of continuous, full-time employment, and every four years thereafter
• Volunteer time off: Two days off for social and civil activism, volunteer work, and/or any related activities, including attending and participating in protests

• Strong judgment in ambiguous environments and a bias toward simple, reliable systems that scale
• Experience building or operating high-throughput API gateways, proxies, or multi-tenant platform services
• Strong production software engineering experience in Python, TypeScript, Go, or a similar language
• Experience with usage metering, cost attribution, capacity planning, or FinOps
• Experience building observability for distributed systems and leading production incident response
• Practical experience with LLM provider APIs, including streaming, long-running requests, retries, timeouts, cancellation, and rate limits
• Experience with authentication, quotas, credential management, tenant isolation, and auditability
• Hands-on experience with Kubernetes, Terraform, CI/CD, and production service ownership
• Experience with data or ML platform systems such as BigQuery, Airflow, streaming pipelines, model serving, or ML observability
• LiteLLM, Portkey, or a comparable multi-provider AI gateway
• VLLM, Modal, Ray, Triton, PyTorch, or GPU-backed serving
• Temporal or another durable-execution system for long-running LLM requests
• Evaluation, batch inference, fine-tuning, RL, or other post-training infrastructure
• Agent runtimes, sandbox infrastructure, MCP, tool use, or coding-agent infrastructure