See how much of this job your resume covers, and what’s missing.
Want a recruiter to go through it line by line?
Get professional reviewQuestions interviewers often ask for this role, with sample answers.
One click makes a copy of your resume for this role, scores it against the posting, and proposes edits line by line. Your default resume is never changed.
Upload your resume and we draft a letter for this exact role, tailored to what it asks for.
The Senior Data Platform Engineer will own and evolve the Azure-based Data Platform, focusing on Apache Airflow orchestration, Databricks governance, and infrastructure automation. The role involves driving platform observability, managing CI/CD pipelines, and collaborating with data product teams to ensure reliable and scalable data architecture.
We are looking for a Senior Data Platform Engineer to help drive the evolution of our Azure-based Data Platform and Lakehouse architecture.
This role is focused on the platform itself. The professional will own and evolve the foundations that data product teams build on: Apache Airflow orchestration (migrating from ADF), Azure resource management (Key Vault, identity and access, networking, storage), Azure DevOps CI/CD and Infrastructure as Code, Databricks and Unity Catalog governance, observability and cost efficiency, while remaining hands-on and close to the teams that use the platform.
\nOrchestration Platform (Apache Airflow & ADF)
Own the Apache Airflow platform: environments, deployment, upgrades, configuration, connections, variables, pools and DAG standards.
Work on the migration from Azure Data Factory to Airflow and the evolution of the Metadata-Driven Orchestration Framework (DAG generation, reusable operators, dependency and scheduling patterns).
Ensure Airflow is reliable and observable: SLAs, alerting, retries, backfills, health monitoring and clear failure ownership.
Azure Resource Management
Manage Azure resources with Infrastructure as Code (Terraform): Databricks workspaces, Data Lake Storage, Data Factory, Key Vault, networking and supporting services.
Own secrets and identity management: Key Vault, service principals and managed identities, secret scopes and rotation.
Own access management: Azure RBAC, Entra ID groups, Unity Catalog grants and ownership, following least privilege, plus access requests and periodic reviews.
Own Azure DevOps: repositories, CI/CD pipelines, branch policies and multi-environment promotion, including deployment validation and Terraform state.
Track cost and governance of Azure resources: tagging, budgets, right-sizing and cost attribution (e.g. query tagging).
Hands-on Data Platform evolution
Contribute to the implementation of the Data Catalog and Data Quality Framework and it’s lineage across the platform.
Drive observability: pipelines that are observable, debuggable and alert-driven, with clear ownership of failures.
Operate the Databricks platform: Unity Catalog structure, cluster and job policies, Auto Loader ingestion patterns, and performance and cost optimization of workloads.
Technical Leadership
Participate in architecture reviews and technical decision-making, stating trade-offs (performance, cost, reliability, complexity) with data.
Define and promote engineering standards: naming conventions, patterns and guidelines, through code reviews and design reviews.
Collaborate with Architects, Product Owners and data engineers in technical maturation of data products.
Build and maintain developer tooling and automation, including AI-assisted engineering workflows (shared agent instructions, skills, automated PR review).
Support & Operations (Run Team Collaboration)
Provide second-line support to the Run/Operations team for complex production topics.
Troubleshoot and resolve platform incidents, failures and performance issues; lead root cause analysis and post-mortems.
Ensure proper handover, documentation and operational readiness of platform changes.
Continuously improve runbooks, monitoring and operational processes.
Technical Skills
6+ years of experience in data engineering, with significant time on platform or infrastructure topics
Strong technical expertise with Python and SQL (advanced), including PySpark and Delta Lake semantics
Deep hands-on experience with Apache Airflow in production: DAG design, deployment, operations, upgrades, and orchestration of Databricks and ADF workloads
Hands-on experience with Azure resource management:
Key Vault, managed identities and service principals, secret management
Access management: Azure RBAC, Entra ID, Unity Catalog permissions
Azure DevOps (repos, pipelines, service connections, environments) and CI/CD for data platforms
ADLS, networking and security basics, Azure Data Factory
Infrastructure as Code (Terraform) and multi-environment promotion
Hands-on experience with Databricks and Unity Catalog (compute, jobs, governance)
Observability and monitoring, alerting and incident management
Solid understanding of Data Lakehouse architecture, Medallion architecture and metadata-driven architecture; awareness of cost management and performance
Soft Skills
Strong problem-solving and analytical thinking
Ability to work in cross-functional teams and to influence without authority
Ownership mindset and attention to detail
Good communication skills with both technical and business stakeholders: leads with the recommendation, quantifies, and is transparent about uncertainty
Stop the endless job search. Our AI finds and applies to the best jobs for you.
Featuring 221,716+ Jobs in Platform Engineer
Answer easy questions
221,716+ jobs across 15+ categories
Get your best job matches
Only hand-screened, legit jobs
Find a remote job faster
No ads, scams, or junk
“I was the first applicant for a remote marketing position that got listed on the company website the same day I applied. Had an interview within 48 hours!”
— Sarah J.
·
Marketing Manager
★★★★★
Verified
