Senior Azure Databricks Architect

Data Elephant is a fast-growing Canadian data and AI consulting firm. We help organizations design, modernize, and scale cloud data platforms using technologies including Databricks, Microsoft Azure, AWS, Power BI, and modern AI and analytics solutions.

We are looking for a Senior Azure Databricks Architect with deep, hands-on experience designing and deploying secure enterprise Databricks environments. This role requires demonstrated experience with Terraform and Azure Private Link - not simply experience working in an environment where these technologies were already configured.

The Opportunity

You will lead the architecture and deployment of secure Azure Databricks environments for enterprise clients. You will work closely with client security, networking, cloud platform, and data engineering teams to design solutions that meet demanding security, connectivity, governance, and operational requirements.

This is a hands-on architecture role. The successful candidate must be comfortable defining the target architecture, making technical decisions, writing and reviewing Terraform, troubleshooting complex networking issues, and supporting implementation through production deployment.

Key Responsibilities

• Design secure, scalable Azure Databricks platform architectures.
• Deploy and configure Azure Databricks workspaces using Terraform and infrastructure-as-code practices.
• Design and implement Azure Private Link and private endpoint connectivity for Databricks
• Configure front-end, back-end, and browser-authentication private connectivity where required.
• Design customer-managed VNet and VNet-injected Databricks deployments.
• Implement Secure Cluster Connectivity and no-public-IP configurations.
• Design private DNS zones, DNS resolution, routing, NSGs, user-defined routes, and firewall rules.
• Integrate Databricks into hub-and-spoke and other enterprise Azure network architectures.
• Work with security and networking teams to troubleshoot DNS, routing, authentication, and connectivity issues.
• Design identity and access controls using Microsoft Entra ID and Databricks Unity Catalog.
• Establish architecture standards and deployment patterns across development, test, and production environments.
• Implement CI/CD processes for Databricks infrastructure and platform configuration.
• Produce architecture diagrams, design decisions, deployment documentation, and operational runbooks.
• Provide technical leadership and guidance to data engineers, cloud engineers, and client stakeholders.

Required Experience

• Significant experience architecting and implementing Azure Databricks environments.
• Hands-on experience deploying Azure Databricks using Terraform.
• Demonstrated experience implementing Azure Private Link and private endpoints for Databricks.
• Experience with VNet injection, customer-managed VNets, Secure Cluster Connectivity, and no-public-IP deployments.
• Strong understanding of Azure networking, including:
  • Private DNS zones and DNS resolution
• Experience implementing hub-and-spoke architectures, Azure firewalls, network security groups, and user-defined routes
• Experience with ExpressRoute or VPN connectivity
• Experience with Terraform resources 
• Experience designing secure connectivity between Databricks and services such as ADLS Gen2, Key Vault, Azure SQL, Event Hubs, or other Azure data services.
• Strong knowledge of Databricks security, identity, access management, Unity Catalog, and service principals.
• Experience establishing CI/CD and infrastructure-as code practices for enterprise data platforms.
• Ability to translate security and business requirements into practical architecture decisions.
• Strong written and verbal communication skills, including experience working directly with clients and technical stakeholders.

Preferred Qualifications

• Databricks architecture or engineering certification.
• Relevant Microsoft Azure certification.
• Experience with Databricks Asset Bundles.
• Experience with Azure DevOps or GitHub Actions.
• Knowledge of data governance, medallion architecture, observability, and cloud cost management.
• Consulting or professional services experience.

What Success Looks Like

You can describe Azure Databricks environments you have personally designed and deployed using Terraform and Private Link. You understand how front-end access, back-end connectivity, browser authentication, private DNS, routing, and no-public-IP cluster connectivity work together - and you can diagnose issues when they do not.

You combine strong architecture judgment with the ability to work hands-on and guide a solution through implementation.

Location

Remote within Canada - only candidates based in Canada will be considered for this role.