Principal Data Engineer

Role Statement:

The Principal Data Engineer provides technical leadership for the design, engineering, implementation, and operation of enterprise-scale cloud-native data platforms. Acting as the senior engineering authority within multidisciplinary delivery teams, the role is responsible for translating solution architectures into secure, scalable, resilient, and maintainable technical solutions that support strategic business outcomes.

Working alongside Solution Architects, Data Architects, Cyber Security, Platform Engineering and delivery teams, the Principal Data Engineer independently leads the engineering design and implementation of complex cloud-native data platforms, data pipelines, APIs and serverless applications. The role is accountable for engineering quality, technical governance, operational excellence, and the adoption of modern engineering practices across multiple programmes.

The role combines deep expertise in AWS cloud technologies, modern software engineering, distributed systems, and enterprise data engineering with strong leadership, mentoring, and stakeholder engagement capabilities. The Principal Data Engineer drives engineering standards, promotes reusable frameworks and automation, and ensures solutions are delivered in accordance with enterprise architecture, security, governance and operational standards.

Key Role Accountabilities:

General

·       Lead the technical design and delivery of enterprise-scale cloud-native data platforms and applications.

·       Provide engineering leadership across multiple delivery teams and programmes.

·       Independently own technical delivery from discovery through implementation, production deployment and operational support.

·       Drive engineering excellence through standards, governance, reusable frameworks and continuous improvement.

·       Evaluate emerging technologies and recommend improvements to engineering capability.

·       Act as the senior escalation point for complex engineering and technical delivery issues.

Leadership & Engagement

·       Provide technical leadership across multidisciplinary delivery teams.

·       Work closely with Solution Architects to convert High Level Designs (HLDs) into detailed Low Level Designs (LLDs).

·       Collaborate with Data Architects, Technical Analysts, Developers, Platform Engineers and Cyber Security teams.

·       Build trusted relationships with business stakeholders and executive sponsors.

·       Provide technical assurance throughout the project lifecycle.

·       Mentor and coach Data Engineers and Software Engineers, supporting capability development across the engineering community.

·       Promote engineering best practice through communities of practice and technical governance forums.

Cloud Platform Engineering

·       Design, build and maintain enterprise AWS cloud platforms using serverless and cloud-native services.

·       Deliver highly available, secure, resilient and cost-effective production platforms.

·       Design event-driven architectures using AWS native services.

·       Engineer scalable application platforms supporting operational, analytical and integration workloads.

·       Design resilient distributed systems capable of supporting enterprise-scale business operations.

·       Optimise cloud infrastructure for performance, reliability and operational efficiency.

Data Engineering & Architecture

·       Design and implement scalable data platforms supporting ingestion, transformation, storage and analytics.

·       Architect batch, streaming and event-driven data pipelines.

·       Design and optimise enterprise data models and data storage architectures.

·       Deliver high-performance ETL/ELT solutions.

·       Implement enterprise integration patterns including APIs, event buses, messaging and microservices.

·       Design data platforms supporting reporting, operational analytics and machine learning workloads.

·       Ensure solutions remain scalable, reusable and maintainable throughout their lifecycle.

AWS Engineering

·       Design and deliver solutions across the AWS ecosystem including:

·      AWS Lambda

·     AWS Glue

·     API Gateway

·      AWS Step Functions

·     AWS Lake Formation

·     IAM

·      Amazon S3

·     Amazon EventBridge

·     CloudWatch

·      DynamoDB

·     SNS

·     AWS Secrets Manager

·      Amazon Redshift

·     SQS

·     AWS KMS

·      CloudFormation or Terraform

·       Apply AWS Well-Architected Framework principles across reliability, security, operational excellence, performance efficiency, sustainability and cost optimisation.

Infrastructure & DevOps

·       Design Infrastructure as Code (IaC) solutions using modern automation frameworks.

• Build and maintain CI/CD deployment pipelines.
• Implement automated testing and deployment strategies.
• Support DevSecOps and DataOps engineering practices.
• Automate infrastructure provisioning and environment management.
• Implement automated release management across multiple environments.
• Improve deployment reliability through engineering automation.

Security, Governance & Compliance

• Apply enterprise security architecture and secure engineering principles.
• Design secure cloud environments aligned with organisational security policies.
• Implement IAM, encryption, network security and secrets management.
• Ensure compliance with GDPR, security standards and organisational governance frameworks.
• Support audit, monitoring and operational compliance activities.
• Ensure engineering solutions satisfy enterprise risk and security requirements.

Operational Excellence

• Design production-ready platforms with comprehensive monitoring and observability.
• Implement logging, metrics, tracing and alerting strategies.
• Drive operational resilience and disaster recovery capability.
• Optimise cloud cost through FinOps best practice.
• Improve platform reliability and operational supportability.
• Lead root cause analysis and continual service improvement initiatives.

Documentation

• Produce detailed Low Level Designs (LLDs) from approved Solution Architect High Level Designs.
• Maintain engineering standards, technical documentation and architectural decisions.
• Produce deployment documentation, operational runbooks and support documentation.
• Document APIs, interfaces and engineering patterns.
• Ensure traceability from requirements through to delivered technical solutions.

Team

• Provide technical mentoring to engineers across delivery teams.
• Perform technical design reviews and code reviews.
• Support engineering recruitment and capability development.
• Establish reusable engineering frameworks and standards.
• Champion continuous learning and innovation.
• Contribute to the strategic direction of the Engineering Practice

Knowledge and Skills

Essential skills and attributes:

·       Extensive experience designing and delivering enterprise cloud-native platforms.

·       Expert knowledge of AWS cloud architecture and serverless technologies.

·       Extensive experience building enterprise data platforms.

·       Expert-level Python development with extensive experience designing, developing and supporting cloud-native, serverless applications, APIs, automation frameworks and enterprise data engineering solutions.

·       Expert understanding of distributed systems and event-driven architectures.

·       Extensive experience designing secure enterprise applications.

·       Advanced knowledge of Infrastructure as Code (IaC).

·       Strong CI/CD automation experience.

·       Experience designing enterprise integration solutions.

·       Excellent SQL and data modelling skills.

·       Experience with Redshift, Glue and modern analytical platforms.

·       Strong understanding of cloud security and governance.

·       Experience delivering highly available production platforms.

·       Strong understanding of observability and operational excellence.

·       Excellent stakeholder engagement and communication skills.

·       Ability to translate business requirements into scalable technical solutions.

·       Experience leading multidisciplinary engineering teams.

·       Strong mentoring and coaching capability.

·       Experience operating within Agile delivery environments.