Senior Software Engineer (Data Platform)
Posted Updated
- As a Senior Data Platform Engineer, you will build and operate the data infrastructure that supports Zeta’s products, analytics, machine learning, and AI systems
- This is a broad platform role. Engineers on the team take ownership across Lakehouse architecture, distributed compute, data quality, governance, lineage, security, access controls, and the operational systems that keep the platform running. Individual focus areas will change with team priorities, and you will contribute across the platform as needed
- The team also develops tooling for type-safe, testable data pipelines and operates Spark workloads on dynamically sized Hadoop clusters backed by EC2 Spot instances. We work closely with data engineers, data scientists, infrastructure teams, and product teams to make data dependable, easy to find, secure to use, and cost-effective to process
- Build and operate Lakehouse, Spark, and distributed data-processing systems
- Improve the performance, reliability, cost, and operational visibility of platform workloads
- Develop capabilities for data quality, governance, lineage, metadata, discoverability, security, and access control
- Build the data context required for trusted reporting, ML models, and AI agents
- Evaluate compute and query engines for read-heavy workloads, assessing cost, performance, reliability, and operational fit
- Assess real-time OLAP technologies and identify appropriate production use cases
- Improve deployment, observability, incident response, developer tooling, and platform operations
- Partner with data scientists and data engineers to improve their experience using the platform
- Contribute fixes and improvements to relevant open-source projects, including Spark, Iceberg, and Airflow
- Mentor engineers through code reviews, pairing, design discussions, and knowledge-sharing sessions
- Working knowledge of AWS services such as EC2, S3, Athena, and IAM
- Experience with Kubernetes, Terraform, Airflow, Trino, ClickHouse, Starrocks, or Pinot is a plus
- Strong experience in Python, Scala, Java, or another JVM language
- Experience with Spark or another distributed data-processing framework
- Familiarity with at least 1 of these areas: Lakehouse architecture, Apache Iceberg, data quality, governance, lineage, metadata, query engines, or real-time OLAP
- Strong analytical, communication, and collaboration skills
- Experience building or operating production data platforms, backend systems, or distributed systems
- Experience improving production reliability through monitoring, alerting, deployment automation, and incident response
- 7+ years of software engineering experience