Azure Data Engineer

Job Description

Your Role:

We are seeking a highly skilled Senior Data Engineer to design, develop, optimize, and maintain scalable data solutions in cloud environments. You will be responsible for building robust pipelines, data processing architectures, and analytics platforms that support Business Intelligence, Machine Learning, and Artificial Intelligence initiatives.

• Design, develop, and maintain scalable, high-performance data pipelines.
• Build ETL/ELT processes using Python, PySpark, SQL, Azure Databricks, and Azure Data Factory.
• Implement integration mechanisms for structured and unstructured data from multiple sources.
• Develop storage and processing solutions based on Data Lake and Lakehouse architecture.
• Ensure data quality, consistency, security, and availability.
• Optimize distributed processing workflows for large volumes of data.
• Implement data governance, monitoring, and observability standards.
• Design and implement solutions using Azure Databricks, Azure Data Factory, Azure Functions, Azure Storage Accounts, Azure Key Vault, and Azure Logic Apps.
• Automate deployments via CI/CD pipelines using Azure DevOps.
• Monitor and resolve incidents related to data processes in production environments.
• Apply security and access management best practices in cloud environments.
• Implement secure authentication, authorization, and secret management mechanisms.
• Collaborate with Data Science teams to prepare, transform, and make data available for analytical models.
• Participate in Machine Learning, Artificial Intelligence, and NLP initiatives.
• Implement and support data pipelines for forecasting and predictive analytics models. 

Your Profile:

• Bachelor’s degree in systems engineering, Computer Science, Software Engineering, Data Science, or a related field.
• Python, PySpark, FastAPI, PyTest.
• Microsoft Azure: Azure Databricks (Databricks Notebooks, Databricks Workflows), Azure Data Factory, Azure Functions, Azure Key Vault, Azure Storage Accounts, Azure Logic Apps, Azure DevOps.
• REST APIs.
• SQL.
• Git. 

Desirable:

• Web Scraping. • Microsoft Excel for exploratory analysis and data validation.
• Data and Analytics Knowledge.
• Design and implementation of modern data architectures.
• Massive data processing and distributed computing.
• Data Warehousing and Data Lakes.
• Machine Learning Engineering fundamentals.
• Business Intelligence and enterprise analytics.
• Forecasting and time-series analysis.
• Credit and financial risk modeling.
• Financial analysis and performance metrics.
• Cash flow analysis and modeling.
• Experience working with Agile methodologies and Jira. 

#LI-DC10

#LI-Remote