Staff Software Engineer (Infrastructure Storage)

• In the world of distributed AI, raw GPU and CPU horsepower is just a part of the story. High-performance networking and storage are the critical components that enable and unite these systems, making groundbreaking AI training and inference possible
• The Lambda Infrastructure Engineering organization forges the foundation of high-performance AI clusters by welding together the latest in AI storage, networking, GPU and CPU hardware
• Our expertise lies at the intersection of:
• High-Performance Distributed Storage Solutions and Protocols: We engineer the protocols and systems that serve massive datasets at the speeds demanded by modern clustered GPUs
• Dynamic Networking: We design advanced networks that provide multi-tenant security and intelligent routing without compromising performance, using the latest in AI networking hardware
• Compute Virtualization: We enable cutting-edge virtualization and clustering that allows AI researchers and engineers to focus on AI workloads, not AI infrastructure, unleashing the full compute bandwidth of clustered GPUs
• We are seeking a seasoned Staff Storage Software Engineer with deep experience designing and deploying storage protocol solutions at scale across object, block, and file paradigms
• This is a unique opportunity to work at the intersection of large-scale distributed systems and the rapidly evolving field of artificial intelligence infrastructure. This is an opportunity to have a significant impact on the future of AI
• You will be building the foundational infrastructure that powers some of the most advanced AI research and products in the world
• Technical Leadership:
• Set technical direction for storage software architecture across the Infrastructure Engineering organization, influencing decisions that span petabyte-scale deployments
• Author and review design documents for new storage systems, protocols, and integrations; raise the technical bar across the team
• Mentor and develop senior engineers, providing guidance on systems design, debugging complex distributed systems issues, and navigating technical tradeoffs
• Serve as a technical anchor for cross-functional initiatives involving storage, networking, compute, and control plane teams
• Represent the storage software team in architectural reviews, roadmap planning, and customer-facing technical discussions where needed
• Execution:
• Design, develop, and maintain high-performance storage systems software with a focus on performance, scalability, reliability, and operational simplicity
• Implement and optimize storage protocol APIs across file (NFS, SMB, Lustre), block (NVMe-oF, iSCSI, Fibre Channel), and object (S3) access patterns
• Develop distributed systems for managing and orchestrating storage resources across multiple solutions and redundant arrays
• Collaborate with hardware and system architects to integrate software with storage solutions including NVMe, GPU-direct storage, and DPU-accelerated data paths
• Troubleshoot and resolve complex issues in production data center environments, including performance regressions, protocol mismatches, and hardware failures
• Contribute across the full software development lifecycle — from requirements gathering and system design through deployment, monitoring, and long-term maintenance
• Build and maintain tooling for storage benchmarking, performance profiling, and capacity planning
• Collaboration
• Work closely with storage software and networking teams to execute cross-functional infrastructure initiatives and new data center deployments, including integration of storage protocols across a variety of on-prem solutions
• Partner with the control plane and Kubernetes teams to meet customer and product requirements for usability, reliability, and telemetry
• Work with the observability team to define, build, and track SLOs/SLIs for storage systems
• Coordinate with Networking, Compute, and Storage Engineering teams to deploy high-performance distributed storage solutions that serve AI/ML workloads
• Partner with the Fleet Engineering team to ensure seamless deployment, monitoring, and ongoing maintenance of distributed storage infrastructure
• Innovate:
• Stay current with the latest research and developments in AI and HPC storage technologies, and bring relevant advances into Lambda’s infrastructure
• Work with the Lambda product team to identify emerging trends in AI inference and training that will shape next-generation storage requirements
• Evaluate and prototype new storage solutions, protocols, and hardware integrations - from open-source distributed filesystems to vendor-specific accelerated storage products
• Optimize storage protocol solutions for AI workloads, including checkpoint I/O for training, high-throughput dataset serving, and latency-sensitive inference pipelines

Benefits

• Health, dental vision
• 401k match
• 5 sick days and 12 paid holidays
• Flexible PTO
• Paid parental, medical, and caregiver leave

• Proven track record designing and operating storage infrastructure at scale (multi-petabyte environments preferred) in production data center or cloud settings
• Background working in high-performance computing, AI/ML infrastructure, or large-scale cloud storage environments
• Experience leading technical projects end-to-end, from architecture through delivery with cross-functional stakeholders
• 10+ years of experience in storage systems engineering, with at least 5 years in a technical lead or Staff+ IC role
• Familiarity with DPDK and SPDK and their role in building high-performance, kernel-bypass storage and networking data paths
• Strong proficiency in one or more low-level systems programming languages: C, C++, Rust, or Go
• Demonstrated ability to write high-performance, concurrent, production-grade systems code and conduct thorough code reviews
• Experience with kernel-level storage drivers, user-space I/O frameworks, or storage daemon development is a strong plus
• Experience implementing or maintaining storage protocol servers or clients in production, not just consuming them
• Familiarity with storage API performance characteristics such as latency, throughput, IOPS and the ability to diagnose and resolve bottlenecks at the protocol level
• Deep hands-on experience with two or more storage protocols: object (S3 or similar), block (iSCSI, Fibre Channel, NVMe-oF), or file (NFS, SMB, Lustre, DAOS)
• Experience with GPU-direct storage or similar zero-copy data paths is a plus
• Familiarity with storage observability tooling — metrics pipelines (Prometheus, Grafana), log aggregation, and tracing in distributed storage environments
• Experience building and operating storage systems with strong reliability expectations: designing for failure, building runbooks, and driving incident response
• Understanding of I/O scheduling, caching layers, write amplification, and related performance tradeoffs
• Experience profiling and tuning storage systems for throughput, latency, and IOPS under real production workloads
• Familiarity with NVMe, NVMe-oF, and RDMA (RoCE or InfiniBand) and their impact on storage system architecture
• Working knowledge of DPUs (e.g., NVIDIA BlueField) and their role in offloading storage and networking data paths
• Comfort working in a physical data center environment — understanding rack-scale infrastructure, storage array hardware, cabling, and failure domains
• Familiarity with tools such as fio, blktrace, perf, eBPF/bpftrace, or equivalent for storage performance analysis
• Experience with NVIDIA BlueField DPUs or SuperNICs for accelerated storage data paths, including GPUDirect Storage implementation
• Deep production experience with enterprise or HPC storage platforms: Vast Data, Weka, NetApp, or IBM Spectrum Scale
• Experience deploying and operating Ceph at scale (100PB+) in an HPC or AI infrastructure environment
• Familiarity with emerging storage technologies such as CXL memory pooling, computational storage, or ZNS (Zoned Namespace) SSDs
• Experience contributing to or maintaining open-source storage projects (e.g., Ceph, DAOS, Lustre, MinIO)