Please mention DailyRemote when applying
AI Summary
Design, develop, and maintain scalable ETL/ELT pipelines and cloud-based data solutions using Azure and Databricks. Collaborate with stakeholders to optimize data workflows and ensure data quality, security, and compliance.
This is a remote position.
Job Summary : We are seeking a skilled and motivated Data Engineer to design, develop, and maintain scalable data pipelines and cloud-based data solutions. The ideal candidate will have strong experience in ETL development, Azure Cloud, Databricks, PySpark, Python, and database technologies, with the ability to build robust data platforms that support analytics, reporting, and business intelligence initiatives.
Key Responsibilities :
Design, develop, and maintain scalable ETL/ELT pipelines for data ingestion, transformation, and processing.
Build and optimize data solutions using Azure Data Platform services and Databricks.
Develop data processing frameworks using PySpark and Python for large-scale datasets.
Implement data integration solutions from multiple structured and unstructured data sources.
Create and manage data workflows, orchestration, and automation processes.
Optimize data pipelines for performance, scalability, reliability, and cost efficiency.
Collaborate with business analysts, data scientists, architects, and stakeholders to understand data requirements.
Ensure data quality, governance, security, and compliance standards are maintained.
Perform data modeling, database design, and query optimization.
Troubleshoot production issues and provide ongoing support for data platforms.
Participate in code reviews, testing, and deployment activities following best practices.
Technical Skills :
Strong experience in ETL/ELT development and data integration.
Hands-on experience with Azure Cloud Services, including:
Azure Data Factory (ADF)
Azure Data Lake Storage (ADLS)
Azure Synapse Analytics (preferred)
Azure SQL Database
Strong experience with Azure Databricks.
Proficiency in PySpark for distributed data processing.
Strong programming skills in Python.
Good understanding of relational databases and SQL.
Experience with database concepts including:
Data modeling
Schema design
Query optimization
Indexing and performance tuning
Experience working with large-scale data processing and cloud-based data architectures.
Preferred Skills :
Experience with Delta Lake and Lakehouse architecture.
Knowledge of CI/CD pipelines and DevOps practices.
Experience with Git and version control systems.
Understanding of data governance and security best practices.
Exposure to streaming technologies and real-time data processing.
Experience with Agile/Scrum methodology.
Automatically Apply to the Best Remote Jobs
Stop the endless job search. Our AI finds and applies to the best jobs for you.
