Data Engineer

Job description

Key Responsibilities

- Design, build, and maintain scalable, reliable data pipelines (batch and streaming).

- Develop and optimise data models, warehouses, and lakehouse architectures.

- Build and automate CI/CD pipelines for data workflows and deployments.

- Ensure data quality, integrity, lineage, and governance across environments.

- Implement proactive monitoring and alerting for pipelines and data infrastructure (CloudWatch, Grafana).

- Manage IaC for consistent, reliable data infrastructure.

- Troubleshoot and optimise pipeline and query performance across environments.

- Lead incident resolution and root cause analysis for data systems.

- Collaborate with analysts, data scientists, and engineering teams, ensuring security, scalability, and operational excellence to understand data requirements and translate them into technical solutions

- Document data architecture, lineage, and workflows

Must have

- 5+ years in data engineering, with solid DevOps/platform engineering exposure.

- Strong SQL and data modelling skills (dimensional, normalised, lakehouse).

- Proficient in Python; working knowledge of Scala, Java, or Go.

- Hands-on experience with data pipeline/orchestration tools (Airflow, DBT).

- Experience with distributed processing frameworks (Spark, Flink, or similar).

- Advanced AWS expertise ( Glue, EMR, Redshift, Athena, S3, Lambda, Kinesis).

- CI/CD tools (GitHub Actions, GitLab CI, Jenkins, CircleCI).

- Infrastructure as Code (CDK, CloudFormation).

- Experience with streaming/event-driven architecture (Kafka, Kinesis, Apache Flink, or Pub/Sub).

- Observability tools: Grafana, CloudWatch.

- Expertise with data warehouse platforms (Redshift, Snowflake).

Desirable

- Experience with containerisation and orchestration (Docker, ECS, Kubernetes).

- Understanding of data governance, cataloguing, and security/compliance (GDPR, lineage tools).

- Experience with GitHub Enterprise, Atlassian APIs, and Slack bots.

- Understanding of cost optimisation strategies in multi-cloud and data warehouse environments.