Design, build, and maintain scalable batch and streaming data pipelines using PySpark and Databricks. Implement data quality frameworks, optimize Spark performance, and manage data lineage for federal client systems.
The Databricks Platform Engineer will deploy and manage Databricks workspaces on AWS using Terraform and implement robust networking and security governance. They will also maintain CI/CD pipelines, monitor platform health, and provide FinOps recommendations to optimize utilization.
