Data Engineer

On behalf of National Crime Agency, we are looking for a Data Engineer Inside IR35 for a 6 month contract based REMOTELY.

As a Data Engineer , your main responsibilities will be:

SC/DV Clearance is an essential requirement for this role, as a minimum you must be willing & eligible to undergo checks. Please note, due to the exceptional requirements of this position (short-term nature of this role and speed at which we require a postholder in situ) preference may be given to candidates who meet all of the essential criteria and hold active security clearance.

Key Responsibilities

• Design, build and maintain scalable data pipelines within Microsoft Azure
• Migrate and transform data from legacy Oracle systems into cloud-based platforms
• Develop and support ETL/ELT processes using Azure Data Factory and Azure Databricks
• Perform data mapping, schema transformation and data modelling activities
• Manage and optimise data movement into Azure Data Lake environments
• Work with Delta Lake/Delta formats to support efficient data storage and processing
• Ensure high levels of data quality, integrity and reliability across all datasets
• Troubleshoot and resolve data-related issues and pipeline failures
• Support both batch and real-time (streaming) data processing requirements
• Produce clear technical documentation, including data dictionaries and metadata records
• Contribute to data governance activities including data lineage, classification and quality controls
• Design secure, scalable and high-performing data solutions
• Work collaboratively with project teams and stakeholders to meet programme objectives
• Deliver knowledge transfer and handover sessions to ensure long-term supportability

Essential Skills & Experience

• Proven experience working as a Data Engineer in large-scale data migration projects
• Strong experience migrating data from Oracle databases into Microsoft Azure
• Hands-on expertise with Azure Databricks
• Experience using Azure Data Factory (ADF)
• Experience working with Azure Data Lake technologies
• Knowledge of Delta Lake / Delta formats
• Strong ETL/ELT development experience
• Data mapping and schema transformation expertise
• Excellent SQL and data integration skills
• Understanding of data governance principles, including:
  • Data lineage
  • Data classification
  • Data quality management
  • Metadata management
• Experience creating and maintaining technical documentation and data dictionaries
• Ability to build secure, scalable and performant data solutions
• Strong stakeholder communication skills

Desirable Experience

• Azure Synapse Analytics
• Microsoft Fabric
• Python and/or PySpark
• Spark-based data processing
• Experience within government, policing, defence or other highly regulated environments
• Active SC Clearance

Active SC/DV Clearance (Please select level of clearance required. For roles not requiring clearance please delete completely and delete this note)

Please be aware that this role can only be worked within the UK and not Overseas.

Disability Confident 

As a Disability Confident employer, CA encourage applications from disabled and neurodivergent candidates. Applications from candidates who identify as having a disability and meet the essential criteria for the role will be prioritised for review. Where application volumes are high, additional role-specific and desirable criteria may be applied as part of the shortlisting process which may include holding active security clearance.

In applying for this role, you acknowledge the following "this role falls in scope of the Off Payroll Working in the Public Sector legislation. Any rates of payment quoted will reflect the gross rate per day for the assignment and will be subject to appropriate taxes and statutory costs. As such the payment to the intermediary and your income resulting from this contract will be different".