Data Engineering Lead (AWS and Azure)

Job description

Leads a team of data engineers delivering pipelines and platform components across AWS and Azure, balancing hands-on technical work with people leadership. Partners with the Data Architect and business stakeholders to keep the team's roadmap aligned with platform architecture and delivery priorities.

Key Responsibilities:

- Lead and mentor a team of data engineers delivering pipelines across AWS and Azure

- Own technical design and implementation decisions for data pipeline architecture

- Establish and enforce engineering best practices: code quality, testing, CI/CD, documentation

- Plan and prioritize the team's roadmap in collaboration with architecture and business stakeholders

- Troubleshoot and resolve complex data pipeline and platform issues

- Drive adoption of orchestration, monitoring, and data-quality tooling

- Partner with the Data Architect to keep delivered pipelines aligned with platform architecture

Requirements:

- 7+ years of data engineering experience, including 2+ years in a technical leadership role

- Hands-on expertise with both AWS (Glue, EMR, Redshift, Lambda, Kinesis) and Azure (Data Factory, Synapse, Databricks)

- Strong background in pipeline architecture, orchestration (Airflow/ADF), and CI/CD for data

- Experience mentoring engineers and setting technical best practices across a team

- Solid SQL and Python/Scala skills, with experience in distributed data processing (Spark)

- Experience with data quality, testing, and monitoring frameworks

- Strong communication and planning skills; comfortable working directly with stakeholders

Nice to Have:

- Experience with dbt or similar transformation frameworks

- Familiarity with infrastructure-as-code (Terraform, CloudFormation, Bicep)

- Prior experience in a consulting or client-facing delivery environment