Data Engineer

Job description

Project Description

The Data Engineer will join a data-infrastructure team responsible for a scalable, robust, self-service analytics platform.

Responsibilities:

- Maintaining an ingestion pipeline processing billions of events daily against a defined SLA

- Supporting teams that build and optimize complex data pipelines

- Identifying platform problems with other engineering teams

- Developing tools and frameworks for the data platform

- Supporting data scientists and analysts with production workloads

- Contributing to ML-platform and streaming initiatives

- Establishing software-engineering and data-development practices

Requirments:

Must have:

- 3+ years of experience in data engineering

- Experience working with an orchestration tool (e.g., Airflow)

- Experience working with a large data set

- Experience building complex ETL pipelines

- Experience working with cloud providers such as GCP, AWS, or Azure

- Strong programming skills in Spark with Scala and Python

- Experience working with CI/CD tools like Jenkins and Git

- Strong understanding of Software Engineering practices and principles

- Excellent problem-solving and communication skills

- Self-motivated and have a proven ability to take the initiative to own the problems that come up and solve them

Nice to have:

- Preferred experience with Kubernetes

- Kafka or another messaging system, streaming platforms

- BigQuery or Redshift data-warehouse administration.