Senior Data Engineer

Senior Data Engineer / Technical Team Lead

Position Summary We are seeking an experienced Senior Data Engineer / Technical Team Lead to design, build, and optimize scalable data platforms and analytics solutions. The ideal candidate will possess strong expertise in Databricks, Azure, AWS, modern data engineering practices, and cloud-native architectures. In addition to hands-on technical responsibilities, the candidate will lead a team of engineers, collaborate with business stakeholders, gather requirements, and drive successful project delivery.

Technical Leadership

· Lead and mentor a team of Data Engineers and support their technical growth.

· Define data engineering standards, best practices, and architectural guidelines.

· Conduct code reviews and ensure adherence to quality and security standards.

· Drive technical design discussions and provide architectural recommendations.

· Collaborate with Solution Architects, Product Owners, and Business Analysts.

Data Engineering & Platform Development

· Design, develop, and maintain scalable data pipelines using Databricks, Spark, and cloud-native services.

· Build and optimize batch and real-time data processing solutions.

· Implement Medallion Architecture (Bronze, Silver, Gold) and Delta Lake best practices.

· Develop ETL/ELT workflows using Azure Data Factory, Databricks Workflows, or equivalent orchestration tools.

· Integrate data from multiple sources including APIs, databases, files, streaming platforms, and SaaS applications.

· Ensure data quality, governance, lineage, and observability across the data ecosystem.

· Optimize performance, scalability, and cloud infrastructure costs.

Stakeholder Management & Requirement Gathering

· Work closely with business stakeholders to understand requirements and translate them into technical solutions.

· Prepare technical specifications, solution designs, and implementation plans.

· Manage project scope, risks, dependencies, and delivery timelines.

· Communicate project status, technical decisions, and risks to leadership teams.

Cloud & DevOps

· Design and implement cloud-native data solutions on Azure or AWS.

· Implement CI/CD pipelines and Infrastructure-as-Code practices.

· Manage cloud security, access control, monitoring, and compliance requirements.

· Support production deployments and troubleshooting activities.

Required Skills & Experience

· 6+ years of experience in Data Engineering and Analytics.

· 3+ years of experience leading data engineering teams.

· Strong expertise in Databricks, Apache Spark, and Delta Lake.

· Hands-on experience with Azure Databricks, Azure Data Factory, ADLS, Event Hub and Key Vault.

· Nice to have experience with AWS services including S3, Glue, Lambda, Redshift, EMR, and Kinesis.

· Strong proficiency in Python, PySpark, and SQL.

· Experience with Kafka, Event Hubs, or Kinesis.

· Experience with Data Warehousing, Data Lakes, and Lakehouse architectures.

· Experience with Git, Azure DevOps, GitHub Actions or Jenkins.

· Strong stakeholder communication, requirement gathering, and Agile delivery experience.

· Experience with Unity Catalog and Databricks governance frameworks.

· Experience with real-time analytics and streaming architectures.

· Exposure to AI/ML data platforms and MLOps.

· Relevant Azure, AWS, or Databricks certifications.

Position Details

Experience: 8–15 Years

Role: Senior Data Engineer / Technical Lead