DataStage Engineer
Please mention DailyRemote when applying
AI Summary
The engineer will analyze legacy IBM InfoSphere DataStage workflows and translate business logic for PySpark implementation on Databricks. They are responsible for designing automated testing frameworks to ensure data integrity and performing comprehensive reconciliation between legacy and cloud environments.
Only Independent Visa candidates
Job Summary
Job Summary
We are seeking an experienced DataStage Migration Validation & ETL Test Engineer to support the migration of legacy IBM InfoSphere DataStage workloads to the Databricks platform. The ideal candidate will possess strong expertise in DataStage architecture, ETL validation, automated testing, SQL, and Python, with the ability to analyze legacy ETL logic and validate equivalent PySpark implementations. This role is critical in ensuring data integrity, accuracy, and successful migration to modern cloud-based data platforms.
Key Responsibilities
- Analyze and interpret complex IBM InfoSphere DataStage Parallel and Server jobs, sequences, and workflows.
- Translate legacy ETL business logic into detailed functional documentation for the PySpark development team.
- Design, develop, and execute automated testing frameworks for large-scale ETL migration validation.
- Perform comprehensive data reconciliation between DataStage and Databricks environments.
- Develop automated validation scripts to verify:
- Data schemas
- Row counts
- Data quality
- Business rule transformations
- End-to-end ETL processing
- Execute regression testing on migrated PySpark pipelines using historical production datasets.
- Identify, analyze, and resolve data discrepancies throughout migration testing.
- Document validation metrics, testing results, and migration KPIs.
- Collaborate with Data Engineers, Architects, QA teams, and business stakeholders to ensure migration quality.
- Provide formal migration validation sign-off prior to production cutover.
Required Skills
- Strong experience with IBM InfoSphere DataStage (Parallel Jobs, Server Jobs, Sequences).
- Experience analyzing legacy ETL workflows and operational metadata.
- Expertise in ETL testing and data validation methodologies.
- Strong SQL skills for complex data analysis and reconciliation.
- Proficiency in Python for automation and validation scripting.
- Experience with XML and JSON parsing.
- Knowledge of automated testing frameworks such as:
- PyTest
- Great Expectations
- Other ETL/Data Validation tools
- Experience performing regression and reconciliation testing for enterprise data migrations.
Preferred Skills
- Exposure to Databricks and PySpark.
- Experience working with modern cloud data warehouse platforms.
- Knowledge of data migration best practices and cloud modernization initiatives.
- Familiarity with Agile development methodologies and CI/CD processes.
Qualifications
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
- 5+ years of experience in ETL development, testing, or data migration projects.
- Prior experience supporting enterprise-scale DataStage modernization initiatives is highly preferred.
Nice to Have
- Experience with Delta Lake or Lakehouse architecture.
- Knowledge of cloud platforms such as Azure, AWS, or GCP.
- Experience with data quality frameworks and automated reconciliation tools.
- Understanding of DevOps and test automation practices.
This is a remote position.
Automatically Apply to the Best Remote Jobs
Stop the endless job search. Our AI finds and applies to the best jobs for you.
