Lead Data Engineer

This is a remote position.

Screening Checklist:
Proficiency in interpreting data transformation logic written in T-SQL and implementing equivalent processes within Databricks
Ability to design and implement data ingestion pipelines using Azure Data Factory (from source to RAW layer)
Basic knowledge of C# and Sql(atleast read the coding, no need to write)
Experience in collecting and analyzing performance metrics to optimize data ingestion pipelines
Competence in performing performance optimizations for Databricks read/write queries as needed
Strong motivation and the ability to provide guidance to other data engineers within the team

Job Overview
We are currently looking for highly accomplished Senior Data Engineers (10+ years of experience)
with deep expertise in Databricks and PySpark to play a key role in an ongoing, large-scale data
transformation initiative. The ideal candidates will bring extensive hands-on experience, strong
architectural understanding, and the ability to lead and influence data engineering practices across
teams.
Key capabilities and expectations include:
• Extensive experience in analyzing, interpreting, and modernizing complex data
transformation logic written in T-SQL, and implementing optimized, scalable equivalents within
Databricks using PySpark.
• Proven expertise in architecting and implementing end-to-end data ingestion frameworks
using Azure Data Factory, managing data flows from multiple source systems through the RAW
and subsequent data layers, with a strong focus on reliability and scalability.
• Strong experience in defining, capturing, and analyzing performance metrics, enabling
proactive monitoring, bottleneck identification, and continuous optimization of data ingestion
pipelines.
• Demonstrated ability to perform advanced performance tuning and optimization of Databricks
read/write operations, including partitioning strategies, caching, file formats, and query
optimization techniques.
• Capability to provide technical leadership and mentorship to data engineering teams, establish
best practices, conduct design reviews, and drive engineering excellence.
• Strong problem-solving skills, a results-oriented mindset, and the ability to collaborate effectively
with cross-functional teams, including architects, analysts, and stakeholders.
This role demands a senior engineering mindset with a balance of deep technical expertise,
architectural ownership, and people leadership to deliver scalable, high-performance data solutions in an
enterprise environment.
Educational qualifications:
Bachelors’s degree or Computer CIENCE