Databricks Platform Engineer
Please mention DailyRemote when applying
AI Summary
The Databricks Platform Engineer will build, maintain, and optimize enterprise data lakehouse infrastructure while ensuring platform reliability and scalability. They will also implement CI/CD pipelines, manage workspace configurations, and provide technical support to data teams.
PCI Professional Services is seeking a Databricks Platform Engineer for an upcoming contract. This role is remote and reports to the Program Manager. The ideal candidate will have strong DevOps and data engineering skills, with expertise in automating Databricks platform operations, ensuring reliability, and enabling data teams to efficiently leverage the platform.
Responsibilities:
- Build, maintain, and optimize the enterprise data lakehouse infrastructure.
- Platform provisioning, monitoring, cost optimization, and providing technical support for Databricks workloads.
- Build and maintain Databricks workspace infrastructure across development, staging, and production environments
- Design, implement, and maintain Databricks platforms in cloud environments (Azure, AWS, or GCP), ensuring scalability, reliability, and cost-efficiency. Manage and optimize Databricks clusters, including autoscaling, resource allocation, monitoring, and troubleshooting.
- Oversee the deployment and configuration of Databricks workspaces, including user management, access controls, and workspace policies.
- Collaborate with infrastructure and DevOps teams to implement CI/CD pipelines for deploying Databricks workflows and notebooks. Monitor and manage performance, logging, and system health of the Databricks platform. Implement robust data security and governance measures in compliance with organizational policies and standards.
- Work with cross-functional teams to troubleshoot and resolve platform-related issues.
- Automate repetitive tasks, such as provisioning, upgrades, and scaling, using Infrastructure-as-Code (IaC) tools like Terraform or CloudFormation.
- Enable data engineers and data scientists by providing support for integrating Databricks with various data sources and tools.
- Stay up-to-date with the latest Databricks features, updates, and best practices to ensure the platform meets evolving business needs.
Minimum Qualifications:
- Bachelor's degree in related discipline
- 5+ years of experience in platform engineering, cloud infrastructure, or IT operations, with 2+ years of hands-on experience managing Databricks environments.
- Public Trust (or higher) clearance eligibility
- Strong knowledge of Databricks administration, including cluster setup, workspace management, and job orchestration.
- Proficiency in Apache Spark for distributed data processing.
- Hands-on experience with cloud platforms (Azure Government, AWS GovCloud, or GCP Public Sector).
- Proficiency in scripting and automation with Python, Bash, or PowerShell.
- Experience with Infrastructure-as-Code (IaC) tools like Terraform, CloudFormation, or ARM templates.
- Solid understanding of cloud networking concepts, such as VPCs, subnets, security groups, and private endpoints.
- Familiarity with federal compliance frameworks, including FedRAMP, FISMA, NIST 800-53, and CMMC.
- Experience implementing data security best practices, such as RBAC, encryption (at rest and in transit), and audit logging.
- Ability to set up and use monitoring and logging tools (e.g., Datadog, AWS CloudWatch, or Azure Monitor) to ensure platform stability.
- Proficiency in diagnosing and resolving cluster performance, job failures, and other technical issues.
Preferred Skills
- Databricks Certified Associate or Platform Administrator certification
- Knowledge of Unity Catalog administration and data governance implementation
- Experience with cost optimization strategies for Databricks, including efficient resource allocation, autoscaling, and workload management.
- Advanced knowledge of performance tuning for Databricks clusters and Spark jobs.
- Implementation of fine-grained access controls and data masking to comply with federal data protection standards.
- Experience implementing single sign-on (SSO), multi-factor authentication (MFA), and data retention policies for secure environments.
- Hands-on experience building automation scripts for platform provisioning, monitoring, and maintenance.
- Familiarity with automation frameworks and tools that integrate with cloud platforms (e.g., AWS Lambda, Azure Logic Apps, Google Cloud Functions).
- Knowledge of ETL/ELT pipelines and experience integrating Databricks with data warehousing solutions like Redshift, Snowflake, or BigQuery.
- Familiarity with real-time data streaming technologies, such as Kafka, AWS Kinesis, or Azure Event Hubs
- Federal cloud certifications such as AWS Certified Solutions Architect – Associate/Professional, Azure Administrator Associate, or Google Cloud Associate Cloud Engineer.
- Prior federal client experience.
PCI Federal (PCIF) and its subsidiaries are an equal-opportunity employer. PCIF does not discriminate on the basis of age, sex, race, national origin, religion, marital status, sexual orientation or identity, Veterans or Disability status.
Preference may be extended to qualified Native American Indian candidates in accordance with applicable federal law.
Automatically Apply to the Best Remote Jobs
Stop the endless job search. Our AI finds and applies to the best jobs for you.
