Senior Data Engineer

Softeta is a software engineering partner for finance, energy, industrial, and other high-stakes sectors. We specialize in building and modernizing backend-heavy, integration-critical systems where speed, accuracy, and reliability are non-negotiable. With 100+ AI and custom software experts across engineering hubs in Lithuania and Poland, we embed ourselves directly into client operations — delivering custom software, automation, and AI where they drive the most value. Clients such as SEB, NordVPN, and CarVertical trust us with systems that serve millions of users globally. We strive for excellence, that’s why we created Cobble – Softeta’s own AI-native development workflow, built by our engineers, that standardizes how developers work with AI agents from requirement to working code.

We’re looking for an experienced Senior Data Engineer to lead the architecture, migration, and optimization of production data pipelines using Airflow, Spark, dbt, BigQuery, and Python. You’ll drive complex pipeline migrations, ensuring identical outputs through robust regression testing, while establishing engineering best practices around code quality, testing, security, and incident management.

Job description:

• Propose and lead the architecture for user-data workflows across Airflow, Spark, dbt, and BigQuery (encryption points, derived tables, backfills, partitions).
• Sequentially migrate existing pipelines behind a custom regression harness to guarantee identical output.
• Establish lasting engineering standards post-migration: code reviews, testing protocols, incident post-mortems, and release checklists.
• Refactor pipelines to eliminate duplicate steps and redundant logic.
• Implement verification jobs to validate end-to-end data lifecycle rules.
• Mentor and pair with an internal data engineer to ensure the team can independently own the migrated pipelines.
• Participate in pipeline maintenance, incident response, and root-cause resolution.
• Document design choices using Architecture Decision Records (ADRs).

Requirements

• Deep hands-on experience with Airflow, Spark, dbt, BigQuery, and Python.
• Proven track record leading a live pipeline migration technically from conception to completion.
• Hands-on experience refactoring production pipelines while verifying identical output.
• Demonstrated success establishing engineering best practices (testing, code reviews, incident resolution) within a team.
• Concrete mentoring or technical leadership experience.

Nice to have:

• Previous experience with encryption at rest, data retention policies, or regulatory compliance (e.g., GDPR).
• Envelope encryption using key management services (GCP KMS), BigQuery AEAD functions, or Tink.
• BigQuery column-level security, policy tags, and Google Cloud Data Catalog.
• Event-driven architectures using Apache Kafka.

Benefits

• Diverse and technically challenging projects.
• Flexible working hours and a hybrid or remote workplace model.
• Flexible schedule and an Agile/SCRUM environment.
• Technical equipment that you can choose.