Lead Data Software Engineer with Databricks with Apache Kafka, Apache Spark, Kubernetes

Job description

We are looking for a Lead Data Software Engineer to drive the migration of an existing data analytics platform, originally built on Azure resources such as Data Factory, Databricks, Event Hub, and Cassandra, to a cloud-agnostic platform deployable on-premises.

Responsibilities

- Implement streaming and batch Spark pipelines running on Kubernetes

- Lead the implementation of objects within the data generator framework

- Oversee deployment and testing across local and development environments

- Drive improvement of the current, actively changing solution

- Conduct bug investigation and resolution

- Perform unit testing to ensure solution quality

- Participate in refinement, planning, and demo sessions

- Guide and mentor team members throughout the implementation phase

Requirements

- 5+ years of experience with Apache Spark, Confluent/Apache Kafka, and Kubernetes

- Proficiency in Python

- Familiarity with the data engineering domain and cloud-agnostic platform migrations

- Capability to dive deeper into new technologies in a short time during active implementation phases

- Ability to work independently with syncs with a team lead

- Strong communication skills and proactiveness

- Showcase of being a hands-on team player and technical leader

- Proficiency in English at a B2+ level

Nice to have

- Knowledge of Java

- Familiarity with Kafka Connect, KSQL, and Kafka Streams

- Skills in Ansible and Argo CD

- Understanding of TDD