Lead Data Software Engineer with Databricks, Apache Kafka, Apache Spark, Kubernetes

Job description

We are seeking a Lead Data Software Engineer to drive the migration of an existing data analytics platform, originally built on Azure resources such as DataFactory, Databricks, EventHub, and Cassandra, to a cloud-agnostic platform capable of on-premises deployment.

Responsibilities

- Implement streaming and batch Spark pipelines running on K8S

- Develop objects of the data generator framework

- Deploy and test solutions on local and development environments

- Enhance and refine the current actively evolving solution

- Investigate and resolve bugs

- Conduct unit testing to ensure code quality

- Participate in refinement, planning, and demo sessions

- Guide and mentor team members while syncing with the team lead

Requirements

- 5+ years of experience with Apache Spark, Databricks, and Kubernetes (K8S)

- Proficiency in Python

- Experience with Confluent/Apache Kafka

- Completed EPAM Data Engineering course or equivalent qualification

- Capability to quickly adapt and dive into the active implementation phase of a project

- Ability to work independently while coordinating with a team lead

- Strong communication skills and a proactive, hands-on, team-player attitude

- English proficiency at B2 level or higher

Nice to have

- Knowledge of Java

- Familiarity with Kafka Connect, KSQL, and Kafka Streams

- Background in Ansible and Argo CD

- Understanding of TDD