Sr Data Engineer – Snowflake

Job Summary

We are seeking an experienced Data Engineer to design, develop, and maintain scalable, cloud-based data platforms for enterprise applications. The ideal candidate will have extensive experience with Snowflake, Databricks, dbt, Apache Spark, and Python, along with expertise in building modern Lakehouse architectures, optimizing data pipelines, and delivering high-quality data solutions. You will work closely with business stakeholders, analytics teams, and AI engineers to enable data-driven decision-making and AI initiatives.

Key Responsibilities

• Design, build, and optimize scalable data pipelines for batch, streaming, and API-based data ingestion.
  
• Develop and maintain enterprise data platforms using Snowflake, ensuring high performance, governance, security, and efficient data sharing.
  
• Build and manage transformation workflows using dbt, including reusable models, testing, documentation, and data contracts.
  
• Develop scalable data processing solutions using Apache Spark, PySpark, and Databricks.
  
• Implement modern Lakehouse (Medallion) Architecture with Bronze, Silver, and Gold data layers.
  
• Design robust ingestion frameworks supporting CDC, batch, event-driven, and API integrations.
  
• Optimize complex SQL queries and large-scale data transformations for performance and reliability.
  
• Build orchestration workflows using Apache Airflow, Azure Data Factory, or equivalent tools.
  
• Implement data quality validation using Great Expectations, dbt tests, or similar frameworks.
  
• Develop CI/CD pipelines for data engineering workflows using Git and automated deployment practices.
  
• Collaborate with Data Scientists, AI Engineers, Business Analysts, and client stakeholders to deliver reliable and scalable data solutions.
  
• Support AI and GenAI initiatives by preparing high-quality datasets and enabling data pipelines for RAG-based applications.
  

Required Technical Skills

• Strong hands-on experience with Snowflake, including architecture, performance tuning, data sharing, security, and governance.
  
• Expertise in Databricks, including Delta Lake, Spark, Unity Catalog, and notebooks.
  
• Production experience with dbt for data transformations, testing, documentation, and data contracts.
  
• Strong programming skills in Python, including PySpark, pandas, and custom data engineering frameworks.
  
• Extensive experience with Apache Spark for large-scale distributed data processing.
  
• Experience with cloud platforms:
  • AWS: S3, Glue, Redshift, Lambda
    
  • Azure: Azure Data Factory (ADF), ADLS Gen2, Azure Databricks, Synapse Analytics
    
• Advanced SQL skills with expertise in query optimization and complex transformations.
  
• Experience implementing CDC, batch, streaming, event-driven, and API-based ingestion pipelines.
  
• Strong understanding of Lakehouse/Medallion Architecture.
  
• Experience implementing data quality frameworks such as Great Expectations, dbt tests, or equivalent.
  
• Hands-on experience with workflow orchestration tools such as Apache Airflow or Azure Data Factory.
  
• Experience with Git, version control, CI/CD pipelines, and automated deployment practices.
  

Requirements

• Strong hands-on experience with Snowflake (Mandatory)
  
• Advanced SQL programming
  
• Python
  
• dbt (Data Build Tool)
  
• Apache Airflow
  
• Data Modeling (Dimensional & Relational)
  
• ETL/ELT Pipeline Development
  
• Spark / PySpark
  
• Cloud Platforms: AWS, Azure, or Google Cloud Platform (GCP)
  
• Git and CI/CD practices