Senior Software Engineer (Observability)

• As a Senior Software Engineer on the Observability team, you’ll help design, develop, and scale the platform used across Roku engineering teams to alert and monitor thousands of services
• You’ll write Go and Rust to build and evolve our own metrics stack on top of open-source technologies, and you’ll work hands-on with tools such as Prometheus, Grafana, Thanos, Loki, Tempo, and OpenTelemetry, optimizing them for performance, reliability, and usability
• AI is central to how you’ll work: you’ll use AI tools and agents daily to build, test, and operate our systems, and help the team keep improving how we engineer with AI. You’ll collaborate with infrastructure and platform teams, automate deployment and configuration, and contribute improvements back to the open-source community
• Building Our Metrics Stack: Design and build high-performance components of our metrics stack in Go and Rust, on top of open-source technologies, to handle terabytes of data daily and hundreds of millions of active series
• AI-Driven Engineering: Use AI tools and agents as part of your daily development and operations work (writing, reviewing, and testing code, automating operational tasks, and accelerating troubleshooting and incident response), and help shape how the team works with AI
• Scaling and Innovating: Enhance and optimize Roku’s observability stack while introducing Roku-specific features and innovations, including AI-assisted capabilities for the engineers who use the platform
• Tooling and Standardization: Work with the Platform team to identify, standardize, and improve observability tools as part of a cohesive solution
• Stability and Performance: Invest in making the observability platform more stable, performant, and reliable for all users
• Automation and Usability: Build automation around observability tools to simplify onboarding for developers (dashboards, alerts, traces, etc.)
• Open Source Contributions: Contribute new open-source tools and improvements back to the CNCF ecosystem
• Collaboration and Engagement: Partner with engineering teams to build solutions that deliver a better out-of-the-box experience and drive innovation
• Technical Leadership: Lead design discussions, mentor engineers and interns, and help shape the platform’s technical direction

Benefits

• Medical, wellness and financial benefits
• Free snacks and access to the company fitness center
• Unlimited paid time off policy
• Work from home opportunities

• This role is ideal for an engineer who’s passionate about systems programming, observability at scale, and AI-driven engineering
• Solid programming experience (typically around 4 years) in Go and/or Rust. Experience with performance-sensitive systems is a strong plus. If you’re strong in one, we welcome your interest in growing in the other
• Deep understanding of Kubernetes and Istio architectures, and the ability to contribute improvements for scalability and integration
• Even if you don’t match every point listed, we’d love to hear from you if you have the drive to dive in, learn, and make an impact
• Expertise in deploying, operating, and extending open-source observability tools at scale, such as Prometheus, Grafana, Thanos, Loki, Tempo, or similar solutions like Cortex, Mimir, or the ELK stack
• Familiarity with open standards such as OpenTelemetry and OpenMetrics
• Education: B.S. or M.S. degree in Computer Science, Engineering, or equivalent experience
• Hands-on experience with AWS, GCP, and/or Azure
• AI-Driven Engineering: Hands-on experience using AI tools and agents in your day-to-day development and operations work, and practical experience building with Large Language Models (for example, tool/function calling, agents, MCP, RAG, and evaluating LLM output)
• Strong ability to engage with customers and stakeholders to deliver tailored observability solutions
• Significant experience (typically around 8 years) in Infrastructure Engineering and/or DevOps, with a focus on observability platforms
• Large-Scale Expertise: Proven experience designing and operating observability platforms that process massive amounts of data (metrics, logs, and traces) to provide actionable insights