Data Engineer II

The ideal candidate combines deep hands-on engineering expertise with strong data modeling and solution design skills and can independently lead complex engineering efforts from requirements through production implementation.

Responsibilities

Advanced Data Engineering
•    Design, build, and maintain scalable data pipelines and data processing solutions using Databricks, Apache Spark, PySpark, Python, and SQL.
•    Develop robust ingestion, transformation, and publishing processes supporting batch and, where appropriate, near-real-time data workloads.
•    Engineer reusable frameworks and patterns that improve development consistency, reliability, maintainability, and speed of delivery.
•    Design solutions capable of processing large and complex datasets while maintaining appropriate performance and scalability.
•    Troubleshoot complex data pipeline, transformation, performance, and integration issues.
•    Optimize Databricks workloads for processing efficiency, reliability, and cost.
Databricks & Lakehouse Engineering
•    Develop enterprise data solutions leveraging Databricks Lakehouse architecture and Delta Lake.
•    Design and maintain structured data layers that move data from raw ingestion through validated, curated, and business-consumption layers.
•    Implement scalable transformation patterns using Spark SQL, PySpark, SQL, and Delta Lake.
•    Apply Databricks engineering capabilities such as Unity Catalog, workflows/jobs, Delta tables, and data governance controls where appropriate.
•    Support performance tuning, workload optimization, monitoring, and production support for Databricks-based solutions.
•    Partner with platform and architecture teams to establish repeatable Databricks engineering standards and development patterns.
Advanced Data Modeling
•    Design scalable logical and physical data models supporting enterprise reporting, analytics, and downstream data products.
•    Develop dimensional models, fact and dimension structures, curated analytical datasets, and reusable semantic-ready data structures.
•    Translate complex operational source data into intuitive business-oriented models.
•    Establish appropriate grain, keys, relationships, historical handling, and business rules within analytical data models.
•    Balance normalization, dimensional modeling, and denormalized analytical structures based on workload and consumption needs.
•    Partner with BI and analytics teams to optimize models for Power BI and other analytical consumers.
Data Quality & Governance
•    Build automated data validation and reconciliation capabilities into engineering pipelines.
•    Establish controls for completeness, accuracy, timeliness, uniqueness, and referential integrity.
•    Implement appropriate monitoring, exception handling, logging, and alerting.
•    Partner with data governance teams to improve metadata, lineage, documentation, and ownership of enterprise data products.
•    Ensure engineering solutions comply with applicable security, privacy, and enterprise data-management standards.
Technical Leadership
•    Serve as a senior technical resource for complex data engineering and modeling initiatives.
•    Lead technical design discussions and recommend appropriate engineering approaches.
•    Perform code and design reviews and help establish development standards and engineering best practices.
•    Mentor data engineers and provide guidance on Databricks, Spark, SQL, Python, data modeling, and performance optimization.
•    Identify opportunities to simplify legacy processes and modernize existing data solutions.
•    Collaborate effectively across onshore/offshore engineering, analytics, architecture, product, and business teams.
Required Qualifications
•    7+ years of professional experience in data engineering, data warehousing, business intelligence engineering, or related disciplines.
•    3+ years of hands-on experience developing production solutions using Databricks and Apache Spark.
•    Advanced proficiency in SQL and strong experience with Python/PySpark.
•    Strong hands-on experience with Delta Lake and modern lakehouse engineering patterns.
•    Demonstrated experience designing and implementing complex ETL/ELT pipelines.
•    Strong understanding of dimensional data modeling, including fact tables, dimensions, surrogate keys, slowly changing dimensions, and analytical data structures.
•    Experience designing solutions for large-volume enterprise datasets.
•    Demonstrated ability to diagnose and optimize complex SQL, Spark, and data-processing workloads.
•    Experience implementing data validation, reconciliation, monitoring, and production support processes.
•    Experience working with source-to-target mappings and translating business requirements into technical data solutions.
•    Strong analytical, troubleshooting, documentation, and communication skills.
•    Ability to independently own technical solutions while collaborating within distributed global teams.
Preferred Qualifications
•    Experience with Databricks Unity Catalog and enterprise data governance.
•    Experience with Databricks workflow orchestration and production deployment patterns.
•    Experience with cloud data platforms and object storage, particularly Microsoft Azure environments.
•    Experience supporting Power BI and enterprise semantic/reporting solutions.
•    Familiarity with CI/CD and automated deployment practices for data engineering.
•    Experience integrating REST APIs, relational databases, SaaS platforms, file-based sources, and other enterprise data sources.
•    Experience modernizing legacy ETL or traditional data warehouse workloads into Databricks/lakehouse architecture.
•    Knowledge of data lineage, metadata management, access controls, and data governance practices.
•    Databricks certifications are a plus.
Leadership & Behavioral Competencies
The successful candidate will demonstrate:
•    Technical ownership: Takes responsibility for solutions from design through production support.
•    Engineering judgment: Selects pragmatic solutions based on scalability, reliability, maintainability, performance, and cost.
•    Problem solving: Can independently investigate complex data and platform issues and drive them to resolution.
•    Quality mindset: Treats data quality, reconciliation, observability, and documentation as core engineering requirements.
•    Collaboration: Works effectively across business, analytics, architecture, BI, and engineering teams.
•    Technical leadership: Raises the capabilities of the broader engineering team through mentoring, standards, reusable patterns, and design guidance.
•    Business orientation: Understands that successful data engineering is measured by the usability and reliability of the resulting data products, not simply by pipeline completion.
What Success Looks Like
A successful Senior Data Engineer will:
•    Deliver reliable, scalable, production-ready Databricks data solutions.
•    Reduce complexity by creating reusable engineering patterns rather than one-off pipelines.
•    Improve the performance and maintainability of enterprise data workloads.
•    Produce well-designed data models that simplify downstream analytics and reporting.
•    Improve data trust through automated validation, reconciliation, monitoring, and documentation.
•    Provide senior-level technical direction to other engineers while remaining highly hands-on.
•    Help advance the Enterprise Intelligence organization toward a more standardized, governed, and scalable lakehouse architecture.
 

Qualifications

Required Qualifications
•    BE with 3-6 years of professional experience in data engineering, data warehousing, business intelligence engineering, or related disciplines.
•    3+ years of hands-on experience developing production solutions using Databricks and Apache Spark.