Staff Backend Engineer (Catalog)

• As AI and data products become business-critical, enterprises face a metadata crisis:
• No unified way to track the complex data supply chain feeding AI systems
• Engineering teams struggling with data discovery, lineage, and governance
• Organizations needing machine-scale metadata management, not just human-browsable catalogs
• This is where infrastructure meets impact
• The metadata layer you’ll build will directly power the next generation of AI systems at massive scale
• Your code will determine how safely and effectively thousands of organizations deploy AI, affecting millions of users worldwide
• We’re looking for an exceptional Python engineer to lead development of DataHub’s Platform framework – the core that connects diverse data systems and powers our metadata collection capabilities
• Scalable, fault-tolerant ingestion systems for enterprise-scale metadata
• Clean, intuitive APIs for our connector ecosystem
• Event-driven architectures for real-time metadata processing
• Schema mapping between diverse systems and DataHub’s unified model
• Versioning systems for AI assets (training data, model weights, embeddings)

Benefits

• Remote friendly
• Carrot Fertility Program
• Parental leave
• Medical, dental, and vision insurance
• Equity
• Work from home and monthly co-working space budget

• Built and maintained online applications serving live traffic at scale (100+ QPS)
• Designed indexing, storage, and data architectures to make large-scale data accessible to online services
• Advanced Python and API design expertise
• Experience with high-scale data processing or integration frameworks
• Designed and scaled distributed systems
• Strong systems knowledge and distributed architecture experience
• Hands-on experience developing in a tight loop with LLMs and applying best practices for scalable LLM development
• Set up monitoring and alerting for services
• Proven track record solving complex technical challenges
• One of Java/Scala/Kotlin/C#/Go - very strong nice-to-have / borderline must-have
• Python/TypeScript/Node.js - nice-to-have
• CI/CD deployment pipelines
• AWS
• Kubernetes/Docker
• Microservice Architecture
• Early-stage startup experience
• Experience fine-tuning LLM-powered applications exposed to end users
• Experience building and maintaining services that make calls to LLMs in order to serve live traffic
• Open-source contributions
• Experience with DataHub or similar metadata/ETL frameworks (Airflow, Airbyte, dbt)
• 8+ years building production-grade distributed systems