Lead Databricks Engineer
Posted Updated
We’re looking for a Lead Databricks Engineer to join our team in London, UK in a hybrid working mode. This role involves designing and leading enterprise-scale Azure and Databricks data platforms to enable advanced analytics and real-time processing capabilities. You will be responsible for platform architecture, governance, cost optimization and ensuring delivery of high-quality solutions that meet critical business needs in a complex, data-driven environment.
Responsibilities
- Lead architecture design and implementation of scalable solutions on Azure Databricks
- Define and enforce platform standards, ensuring compliance with data governance and security policies
- Build and optimize Databricks streaming workloads, including Structured Streaming and Delta Live Tables
- Collaborate with cross-functional teams to translate requirements into robust technical solutions
- Drive platform performance tuning, cost optimization and monitoring strategies for cloud workloads
- Provide leadership across the full engineering lifecycle, including solution delivery and production support
- Mentor and guide developers, establishing best practices for data engineering on Azure and Databricks
- Participate in technical decision-making and review designs for scalability and maintainability
- Implement observability for critical pipelines and maintain quality assurance standards
- Engage directly with stakeholders to ensure alignment on strategic platform initiatives
Requirements
- Minimum 8+ years of experience in data engineering, including 3+ years in a tech lead capacity
- Strong hands-on expertise in Databricks for enterprise-scale workloads
- Proficiency in PySpark, Spark Structured Streaming, and Delta Lake
- Solid programming skills in Python and demonstrated SQL performance optimization experience
- Deep understanding of Azure Data Platform services and cloud-native engineering patterns
- Proven experience implementing data governance, quality management and observability frameworks
- Ability to manage complex data pipelines for batch and streaming use cases in mission-critical environments
- Strong communication and leadership skills to work with distributed teams and senior stakeholders
Nice to have
- Knowledge of Delta Live Tables (DLT) and advanced workflows in Databricks
- Familiarity with CI/CD for data engineering using GitHub or Azure DevOps
- Experience designing infrastructure-as-code solutions for Azure environments
- Understanding of data mesh or lakehouse principles for large-scale architectures
- Background in financial trading or capital markets data domains
Benefits
- EPAM Employee Stock Purchase Plan (ESPP)
- Protection benefits including life assurance, income protection and critical illness cover
- Private medical insurance and dental care
- Employee Assistance Program
- Competitive group pension plan
- Cyclescheme, Techscheme and season ticket loans
- Various perks such as free Wednesday lunch in-office, on-site massages and regular social events
- Learning and development opportunities including in-house training and coaching, professional certifications, and courses
- If otherwise eligible, participation in the discretionary annual bonus program
- If otherwise eligible and hired into a qualifying level, participation in the discretionary Long-Term Incentive (LTI) Program