Assess and re-engineer legacy IBM DataStage ETL jobs into modern Databricks Spark pipelines on AWS. Manage the full development lifecycle including testing, production deployment, migration cutover, and technical documentation.
This is a remote position.
Onsite/Hybrid/Remote: Remote
Duration: 12 months
Rate Range: $68 in W2
Work Authorization: GC and US Citizens Only
Must Have:
IBM DataStage development
ETL/ELT migration and modernization
Databricks and Apache Spark on AWS
PySpark, Python, SQL, and Shell scripting
Delta Lake, Unity Catalog, and Photon
AWS Glue, Lambda, and Redshift
CI/CD and automated ETL testing
PyTest and XML/JSON parsing
Responsibilities:
Assess existing ETL/ELT jobs, databases, and data warehouses for migration readiness.
Re-engineer legacy IBM DataStage jobs using Databricks Spark on AWS.
Develop, test, and deploy scalable cloud-based data pipelines.
Perform parity checks, unit testing, functional testing, UAT, regression testing, and performance testing.
Validate schemas, data quality, pipeline performance, and migration accuracy.
Build CI/CD processes for automated development, testing, and deployment.
Support migration cutover, production deployment, and hypercare.
Coordinate application redeployment and end-to-end testing.
Decommission legacy DataStage jobs and related artifacts.
Prepare technical documentation and operational readiness materials.
Conduct knowledge-transfer and training sessions for engineering and business teams.
Qualifications:
Strong hands-on experience developing and maintaining IBM DataStage jobs.
Proven experience modernizing large-scale ETL/ELT and data warehouse workloads.
Experience building Databricks and Spark pipelines in an AWS environment.
Advanced programming skills in PySpark, Python, SQL, and Shell scripting.
Experience with Delta Lake, Unity Catalog, and Databricks Photon.
Experience with GitLab or Azure DevOps for CI/CD.
Knowledge of automated testing frameworks and ETL validation methods.
Experience supporting production cutovers, stabilization, and application teams.
Nice to Have:
Experience with AWS Glue, Lambda, and Redshift.
Experience with JIRA and Agile delivery.
Knowledge of data governance and cloud security standards.
Experience creating operational documentation and delivering technical training.
“I was the first applicant for a remote marketing position that got listed on the company website the same day I applied. Had an interview within 48 hours!”
— Sarah J.
·
Marketing Manager
★★★★★
Verified
