Inference Software Engineer

• Support porting state-of-the-art models to our architecture. Help build programming abstractions and testing capabilities to rapidly iterate on model porting
• Build, enhance, and scale Sohu’s runtime, including multi-node inference, intra-node execution, state management, and robust error handling
• Optimize routing and communication layers using Sohu’s collectives
• Utilize performance profiling and debugging tools to identify bottlenecks and correctness issues

Benefits

• Competitive Salary and Equity - We offer a competitive salary and equity package that aligns with industry standards
• Benefits - Your wellbeing is important to us. We offer unlimited PTO, life insurance, and a 401(k)
• Full Insurance Coverage - Our insurance includes full medical, dental, and vision with 100% of premium covered

• Understanding of performance-sensitive or complex distributed software systems like Linux internals, accelerator architectures (e.g. GPUs, TPUs), Compilers, or high-speed interconnects (e.g. NVLink, InfiniBand)
• Ported applications to non-standard accelerator hardware or hardware platforms
• Familiarity with PyTorch or JAX
• Proficiency in C++ or Rust
• Developed low-latency, high-performance applications using both kernel-level and user-space networking stacks
• Deep understanding of distributed systems concepts, algorithms, and challenges, including consensus protocols, consistency models, and communication patterns
• Solid grasp of Transformer architectures, particularly Mixture-of-Experts (MoE)
• Built applications with extensive SIMD (Single Instruction, Multiple Data) optimizations for performance-critical paths