Principal Software Engineer (Orchestration)
Posted Updated
Summary
Principal Software Engineer leading the orchestration platform (Airflow on EKS) that schedules 8,000+ daily data workflows for Roblox's 200M+ user platform, focusing on scaling, reliability, and developer tooling.
- Roblox’s data platform powers analytics, ML, and product decisions for a platform serving 200M+ daily active users. As a Principal Software Engineer in the Data Platform Orchestration team, you will be the technical leader for the system that schedules and runs the company’s data workloads
- Our team owns Apache Airflow and the Kubernetes (EKS) infrastructure underneath it, running 8,000+ DAGs every day, most of them Spark jobs. You will set the long-term architecture for how Roblox orchestrates data at scale, and you will be hands-on in making it faster, more reliable, and easier for hundreds of engineers to use
- Define the Orchestration Strategy: Own the multi-year architecture for Airflow and the surrounding platform, including scheduler and executor design, multi-tenancy, upgrades and migrations, and where the platform should evolve next
- Scale and Harden Airflow: Find and remove scaling limits across the scheduler, metadata database, executors, and DAG parsing. Make sure thousands of daily DAGs run with predictable latency and strong SLAs
- Own Orchestration on Kubernetes: Lead the design and operation of our EKS clusters for orchestration and Spark workloads, including autoscaling, resource isolation, cost efficiency, and deployment automation
- Build Platform Tooling for Internal Engineers: Design the APIs, custom operators and providers, libraries, and self-service tooling that let data teams build, deploy, and monitor pipelines safely
- Drive Observability and Reliability: Establish SLOs, alerting, and diagnostics for pipelines and platform components. Lead incident reviews and fix the root causes in code, not just in runbooks
- Lead Across Teams: Work with data platform, governance, data science, and product engineering teams to align roadmaps, guide technical decisions, and build consensus on complex initiatives
- Raise the Engineering Bar: Set standards for code quality, design, and operational readiness. Mentor staff, senior, and mid-level engineers
- Data Workload Familiarity: Working knowledge of how Spark jobs run on Kubernetes (Spark on K8s, spark-operator, or EMR on EKS) and how orchestration interacts with them
- B.S. equivalent in CS or sufficient experience
- 10+ years of experience designing, building, and operating production-grade, large-scale distributed systems
- Workflow Orchestration Depth: Deep, hands-on experience designing, running, and scaling a production data-related workflow orchestrator or job scheduler (thousands of workflows or DAGs a day). Apache Airflow is ideal, but experience with an equivalent system (Dagster, Prefect, Temporal, or a large in-house scheduler) is also welcome. You understand scheduler and executor design, metadata store performance, task isolation, retries and idempotency, and safe upgrades and migrations
- Software Engineering Foundations: Strong proficiency in Python, plus Go or Java, with a record of writing clean, well-tested platform and tooling code. You are a software engineer who builds and improves systems, not someone who mainly handles tickets and operations
- Kubernetes Expertise: Proven experience managing and scaling production Kubernetes clusters on AWS (EKS), including Helm, autoscaling (Karpenter or Cluster Autoscaler), and GitOps-style deployment
- Extreme Ownership: You define the problem when requirements are unclear, unblock dependencies, and see complex work through to stable production
- Cross-Organizational Technical Leadership: A track record of influencing direction across a large engineering organization and driving multi-quarter projects from concept to production
- Platform Mindset: Experience building platforms used by other engineers, with strong instincts for APIs, observability, reliability, and developer experience
- Experience with other orchestration systems (Dagster, Temporal, Argo Workflows, Flyte) and the judgment to know when each fits
- Contributions to Apache Airflow or related open-source projects (Airflow providers, Helm chart, spark-operator, Karpenter, Argo CD)
- Experience with Terraform or other infrastructure-as-code, Prometheus/OpenTelemetry, and cost optimization or chargeback for shared platforms
- Experience operating infrastructure at consumer-internet scale