Senior Data Platform Engineer

Job description

Who we are

DigiCert is a global leader in intelligent trust. We protect the digital world by ensuring the security, privacy, and authenticity of every interaction. Our AI-powered DigiCert ONE platform unifies PKI, DNS, and certificate lifecycle management, to secure infrastructure, software, devices, messages, AI content and agents. Learn why more than 100,000 organizations, including 90% of the Fortune 500, choose DigiCert to stop today’s threats and prepare for a quantum-safe future at www.digicert.com

Job summary

We are looking for a Staff Platform Data Engineer to lead the vision, design, architecture, and scaling of the foundational data and ML infrastructure that powers analytics, reporting, and machine learning across DigiCert. This role sits at the intersection of data engineering and ML platform work — you will own the systems, pipelines, and tooling that data scientists, analysts, and engineers rely on every day. You bring deep expertise in Databricks, a strong engineering mindset, and hands-on experience building and maintaining ML pipelines in production. You will partner closely with data science, analytics, product, and business teams to deliver a platform that is reliable, governed, and built to scale.

What you will do

- Architect and set up Databricks workspaces, AWS VPCs, and underlying network configurations to understand, establish, and secure the end-to-end data flow

- Communicate and collaborate closely with cross-functional engineering, security, and infrastructure team members to align on network topologies and successfully execute platform deployment

- Design, build, and maintain scalable data and ML pipelines using Python and SQL, processing large-scale datasets across batch and streaming workloads

- Own and evolve core platform infrastructure on Databricks — including Delta Lake table architecture, Unity Catalog governance, Databricks Workflows orchestration, and compute optimization

- Build and maintain end-to-end ML pipelines: feature engineering, model training pipelines, experiment tracking (MLflow), and model deployment/serving infrastructure

- Collaborate with data scientists to operationalize models — bridging the gap between experimentation and production-grade ML systems

- Define and enforce data platform standards: ingestion patterns, data modeling conventions, medallion architecture (Bronze/Silver/Gold), and pipeline reliability practices

- Implement data quality, observability, and monitoring frameworks to ensure platform health and data trustworthiness

- Optimize pipelines for performance, cost, and reliability at scale using Spark and PySpark

- Evaluate, integrate, and govern new platform tooling and data sources within the Databricks ecosystem

- Contribute to architectural decisions and help drive the long-term data platform roadmap

- Participate in code reviews, technical design discussions, and engineering standards

- Mentor junior engineers and elevate overall platform and data engineering practices

- Document platform architecture, pipeline design, and operational runbooks

What you will have

- 10+ years of experience in data engineering, data platform, or ML engineering roles

- Strong proficiency in Python and SQL, with a track record of building production-grade data pipelines using both

- Expertise in managing Databricks workspaces, Delta Lake, Unity Catalog, and Spark/PySpark optimization

- Proven background configuring cloud networking concepts (AWS VPCs, subnets, route tables, security groups, private link endpoints) to secure platform data flow and communication

- Hands-on Databricks expertise: Delta Lake, Unity Catalog, Databricks Workflows, PySpark, and the Databricks ecosystem broadly

- Experience building and maintaining ML pipelines in production — feature engineering, training pipelines, experiment tracking, and model deployment

- Familiarity with MLflow or comparable experiment tracking and model registry tools

- Strong understanding of data modeling, dimensional design, and analytics-friendly data architecture

- Experience with batch and incremental/CDC pipeline patterns

- Proficiency with Git, version control, and CI/CD practices for data and ML workflows

- Strong engineering judgment — you think about reliability, maintainability, and cost, not just correctness

- Clear communication and comfort working with both technical and non-technical stakeholders

Nice to have

- Experience with streaming or near real-time pipelines (Kafka, Kinesis, Spark Structured Streaming)

- Familiarity with feature store platforms (Databricks Feature Store, Feast, or Tecton)

- Experience with LLM pipelines, RAG architectures, or AI/BI tooling (Genie, AI Functions)

- Knowledge of data quality and observability tooling (Great Expectations, Monte Carlo, etc.)

- Exposure to dbt or similar SQL-based transformation frameworks

- Infrastructure-as-code experience (Terraform, Databricks Asset Bundles)

- Experience working in Agile or Scrum environments

- Prior experience mentoring engineers or shaping platform standards

Benefits

- Generous time off policies

- Top shelf benefits

- Education, wellness and lifestyle support

To protect candidate information and maintain a secure hiring process, all applications must be submitted through our careers portal. Resumes or CVs sent directly via email will not be reviewed or considered.

#LI-SD1

REMOTELY.LIVING INSIGHTS

Salary insights

We're still collecting salary data for this role and region. Once we have enough comparable job listings, we'll provide a rough market estimate.