Senior Data Engineer (Python/Spark/Azure / MS Fabric)

Job Description

Time Zone Requirement

Resources must be able to provide overlap with the customer's US Eastern Time working hours, including availability through approximately 4:30 PM ET.

About Client

Our client is expanding its data platform team as part of its ongoing data modernization initiatives. They are looking to add two hands-on Data Engineers to support the evolution of the Microsoft-based analytics and data platform.

Role Overview

Senior Data Engineer with production experience designing, optimizing, and operating large-scale Spark workloads on Microsoft Fabric or Databricks.

Strong in Delta Lake performance (liquid clustering, OPTIMIZE, file sizing, incremental merges), Spark UI-driven troubleshooting (skew, spill, OOM, shuffle), and Fabric capacity economics (CU-seconds, bursting, smoothing, throttling stages, pool and node sizing). Builds reliable, cost-aware pipelines that stay inside capacity limits while meeting SLAs for both interactive and batch workloads.

Also develops and maintains internal Python libraries and reusable Spark modules that standardize ingestion, governance, and optimization patterns across teams. 

Must Have

Core Technical Strengths

• Spark load optimization & troubleshooting: Stagelevel diagnosis (task duration histograms, shuffle skew, spill), AQE, broadcast vs shuffle joins, predicate pushdown, salting only when proven, driver vs executor memory pressure, Python UDF cost.
• Delta Lake & Fabric Lakehouse: Liquid clustering vs classic partitioning + Z-order, target file size, small-file mitigation, Vorder for Direct Lake, statistics, VACUUM retention vs time travel, idempotent incremental MERGE patterns with watermarks and quarantine.
• Fabric capacity & compute: CU-second accounting, 10 min / 60 min / 24 h throttling windows (overage protection -> interactive delay -> interactive rejection -> background rejection), carryforward and burndown, starter vs custom vs custom live pools, node sizing (Small through XX-Large) and its effect on max node count, auto-scale and dynamic allocation, single node resource sharing, CU budgeting across concurrent jobs.
• Operational design: Separating interactive, scheduled, and heavy batch workloads; hard caps on pool max nodes; capacity isolation strategies; monitoring via Capacity Metrics app and Spark resource usage UI; maintenance pipelines (OPTIMIZE timing, statistics).
• Python library development: Designs, packages, and versions internal Python libraries and Spark utility modules (ingestion frameworks, Delta metadata/governance helpers, role and security managers, monitoring utilities). Emphasizes production readiness - type hints, logging, error handling, configuration-driven behavior, unit/integration testing, and clear READMEs - so notebooks stay thin and teams share consistent, maintainable code.

Working Style

• Starts with measurement (Spark UI, Capacity Metrics, file-level Delta stats) before changing infrastructure. Prefers code and configuration changes that reduce shuffled bytes and future CU-second debt over "make the pool bigger."
• Documents pool sizing rationale, throttling risks, and maintenance expectations so the next engineer can operate the system.
• Comfortable explaining capacity behavior to both engineering and platform teams.
• Built and iterated production Python libraries and notebook modules used across multiple Fabric workspaces, including standardized Delta ingestion classes (with governance metadata and TBLPROPERTIES support), OneLake data-access role management utilities, and reusable optimization/maintenance helpers. Focused on thread-safety, validation, secrets handling, and documentation so the libraries could be adopted by other engineers without constant support.

What You Will Love About Working Here

We care about all our employees and want them to feel as comfortable as possible. That's why we offer them health insurance from the first days, regardless of the probationary period.

The gift from the company - Christmas holidays from 25 December to 31 December.

Сooperation with Superhumans center and Veteran HUB. Capgemini Engineering has supported the launch of psychological rehabilitation department of Superhumans. Our team also donated over UAH 500 000 prosthetics for three Ukrainian defenders. Currently, we support psychological counseling provided by the Veteran Hub, and we have implemented an internal policy making the company friendly to military and veterans with the assistance of the Hub.