Senior / Staff Software Engineer (AI Platform)

• We’re looking for Software Engineers at Senior / Staff levels who combine strong software and distributed systems fundamentals with experience operating AI systems in production
• You’ll build the infrastructure that lets teams create, experiment with, evaluate, and operate LLM and agentic workloads at significant scale - from synthetic data and evaluation pipelines to simulation environments, orchestration systems, and LLM infrastructure
• You’ll work on systems where correctness is not defined by a single deterministic output. Instead, you’ll build the infrastructure needed to understand behavior across models, prompts, tools, environments, and multi-step trajectories, and to continuously improve those systems through experimentation and evaluation
• We are looking to grow our team of AI Engineers, and are hiring at multiple levels
• Design and build infrastructure for running large-scale agentic workloads, including multi-step agents interacting with tools, external services, sandboxes, and simulated environments
• Build scalable synthetic data generation and automated labeling systems that allow teams to create, refine, and evaluate high-quality training and evaluation datasets
• Design evaluation infrastructure for measuring AI system behavior across models, prompts, tools, environments, and multi-step trajectories - including reproducible experiments, benchmark execution, regression detection, and continuous evaluation
• Build orchestration and distributed compute systems for running thousands to millions of AI experiments and simulations reliably across heterogeneous compute environments
• Develop infrastructure for agent simulation environments, including environment provisioning, isolation, lifecycle management, and scalable execution
• Build and operate LLM infrastructure for routing, rate limiting, retries, caching, provider failover, cost attribution, and efficient execution across multiple model providers
• Instrument agent and model workloads so failures are observable and debuggable - capturing traces, model interactions, tool calls, environment state, evaluation results, latency, reliability, and cost
• Design systems that make non-deterministic workloads reproducible and measurable, allowing engineers to compare experiments, diagnose behavioral regressions, and understand why an agent succeeded or failed
• Improve the developer experience for AI experimentation by building APIs, SDKs, workflow abstractions, and tooling that make it easy to move workloads from local development to large-scale production execution
• Collaborate with research, product, and engineering teams to turn experimental AI workflows into reliable, reusable platform capabilities

Benefits

• Health: Snorkelers and their dependents are covered by comprehensive medical, dental, and vision plans.
• Wellness: All Snorkelers get a yearly wellness stipend to use on anything related to health and well-being.
• PTO and rest days: We provide unlimited personal time off so you can find your work-life balance. Plus, two company-wide rest days per quarter in addition to company holidays.
• One global team: From our headquarters in Redwood City, CA, to regional offices in San Francisco and New York, we work together as one global team.
• Events: Get to know your fellow Snorkelers outside of work! We host regular events and learning opportunities for our teams across our offices and virtually to connect, have fun, and grow.
• Family leave: Snorkel offers generous parental leave for both birthing and non-birthing parents based on eligibility.

• Track record of leading complex engineering initiatives, influencing stakeholders, and delivering measurable impact
• Fluency with modern AI and developer tooling and a willingness to rapidly evaluate and adopt new models, frameworks, infrastructure, and techniques as the ecosystem evolves
• Strong background in distributed systems and cloud platforms (AWS preferred), including compute orchestration, storage, networking, isolation, and failure handling
• Strong understanding of production system fundamentals - observability, telemetry, reliability, performance, debugging, incident response, and cost management
• Experience operating non-deterministic AI or ML workloads in production or at significant scale — you are comfortable reasoning about behavior across models, tools, environments, and multi-step execution
• Ability to work in a fast-paced environment with strong technical communication skills
• Experience building infrastructure for experimentation, evaluation, model development, synthetic data, agentic workflows, training, inference, or production ML systems
• Experience with workflow or distributed execution frameworks such as Prefect, Airflow, Dagster, Ray, Kubernetes, or similar systems
• Strong proficiency in Python and experience building production-quality APIs, services, and developer tooling
• Ability to reason about AI system quality beyond traditional service metrics, including evaluation design, experiment reproducibility, behavioral regressions, and model or agent variability
• 5+ years building production software systems, with experience in AI/ML infrastructure, ML platforms, distributed systems, data platforms, or backend infrastructure
• Experience building or operating LLM or agent infrastructure, including model gateways, agent runtimes, tool execution, tracing, or multi-agent systems
• Experience with synthetic data generation, automated labeling, data refinement, or dataset quality systems
• Experience building evaluation or experimentation platforms for LLMs, agents, or other probabilistic systems
• Experience building reinforcement learning environments, agent simulations, benchmarks, or other environment-based evaluation systems
• Experience running large-scale distributed AI workloads across containers, Kubernetes, serverless compute, sandboxes, or heterogeneous compute environments
• Experience with LLM observability, tracing, prompt/version management, token and cost attribution, rate limiting, caching, or multi-provider routing
• Experience designing isolation and sandboxing infrastructure for executing model-generated code or tool calls safely
• Experience in hyper-growth startup environments or scaling engineering organizations
• Experience building shared AI platform libraries or SDKs consumed by multiple teams, including versioning, backwards compatibility, and migration support
• Prior experience as a Tech Lead, Team Lead, or hands-on Engineering Manager