Senior/Lead Data Engineer

Job description

We are looking for a Senior/Lead Data Engineer to build telemetry and cost-attribution data pipelines that map real-time accelerator utilization to workloads and teams.

Responsibilities

- Build BigQuery/Dataflow pipelines ingesting TensorCore utilization, FLOPs, and network saturation metrics

- Develop cost-attribution dashboards mapping accelerator spend to workloads/teams

- Design reliable, high-throughput data pipelines (Airflow, Cloud Composer, Dataflow)

- Manage complex data ingestion and preprocessing for telemetry data

- Advise on cloud storage architecture for large-scale telemetry data

Requirements

- Strong GCP data stack experience (BigQuery, Dataflow, Cloud Composer)

- Proficiency in SQL and pipeline orchestration tools (Airflow)

- Experience building distributed, high-throughput data pipelines

- Understanding of AI/ML data requirements

- Advanced knowledge of cloud storage systems

Nice to have

- Experience with real-time streaming telemetry systems

- FinOps/cost-attribution dashboard design experience