Member of Technical Staff (Pretraining Systems)
Posted
- As a Research Engineer on the Pretraining Systems team, you will design and operate the distributed infrastructure that trains Magic’s long-context models at scale
- This role focuses on large-scale model training across massive GPU clusters. You will work at the boundary between deep learning and distributed systems, ensuring that training runs are performant, reliable, and reproducible under extreme scale
- Magic’s long-context models create non-trivial systems challenges: sustained memory pressure, communication overhead across thousands of devices, long-running jobs that must survive failures, and efficient sequence packing under hardware constraints. You will own the systems that make large-scale pretraining stable and fast
- Scale distributed training across large GPU clusters (data, tensor, pipeline parallelism)
- Optimize communication patterns and gradient synchronization
- Improve checkpointing, fault tolerance, and job recovery systems
- Profile and eliminate performance bottlenecks across compute, networking, and storage
- Improve experiment reproducibility and orchestration workflows
- Increase hardware utilization and training throughput
- Collaborate with Kernels and Research to align model architecture with systems realities
Benefits
- Retirement savings: 401K with 6% salary matching
- Healthcare: Health, dental and vision insurance for you and your dependents
- Time off: Unlimited paid time off to support balance in your work and life
- Office catering: Chef-prepared lunch and dinner
- Wellbeing: Access to mental health, financial wellbeing, and telehealth programs
- Salary and equity: Competitive salary with significant equity compensation
- Deep understanding of parallelism strategies and performance trade-offs
- Track record of improving performance or reliability of large-scale systems
- Experience training large models in multi-node GPU environments
- Experience debugging cross-layer issues in production ML systems
- Strong software engineering and distributed systems fundamentals
- Strong ownership mindset and ability to operate critical infrastructure
- We value quick learning and grit just as much as skill and experience