Data Engineer

About DataVue

DataVue provides consumer credit data, audience solutions, and predictive analytics to lenders and other businesses. Our data environment spans more than one billion records, with individual source files exceeding 200 million records and some datasets containing more than 4,000 attributes.

The role

We are looking for a hands-on Data Engineer. You will build and operate workflows that ingest, cleanse, match, profile, and integrate large datasets, making them reliable and usable for customer delivery, analytics, and modeling.

Major feeds refresh approximately every four to six weeks. Between refreshes, you will improve automation, onboard sources, maintain data dictionaries, investigate quality issues, and support production data requests. Database responsibilities focus on tables, processing logic, and performance optimization; infrastructure administration is handled separately.

What you will do

Data pipelines and automation

• Build and maintain Python and Spark ETL/ELT workflows in AWS. Manage source mappings, schema changes, full and incremental loads, reconciliation, and recovery from failed processing.
• Automate scheduling, dependencies, logging, alerts, retries, and safe reruns. Optimize runtime, memory use, and processing costs; maintain version-controlled code, tests, and runbooks.

Data cleansing and matching

• Integrate CASS address processing and other cleansing steps. Develop match keys, crosswalks, and record-linkage logic connecting consumers, households, addresses, and properties across sources.
• Handle duplicates, conflicting values, missing identifiers, and ambiguous matches. Monitor match quality and coverage, prevent unintended record multiplication, and maintain consistent identities across refreshes.

Attribute profiling and data quality

• Build configurable profiling across thousands of attributes. Measure meaningful population rates excluding blanks and placeholder codes, validity, distinct values, distributions, ranges, percentiles, and concentration as appropriate.
• Monitor drift between deliveries and assess coverage within relevant populations. Establish quality checks and exception reports, investigate discrepancies, and validate refreshed data before production release.
• Maintain data dictionaries, source lineage, definitions, and quality findings. Help technical, modeling, sales, and operations teams understand attribute usability and limitations.

Database development and data operations

• Design and maintain staging, integrated, and downstream tables. Optimize SQL, large joins, schemas, indexing, partitioning, and data distribution as appropriate to the platform.
• Support customer orders, audience extractions, suppressions, match-and-append requests, and modeling datasets as priorities require. Apply established controls for sensitive consumer data.

Historical data and derived attributes

• Contribute to historical data capabilities using Apache Iceberg or comparable technology, with potential petabyte-scale requirements. Support historical extracts for modeling and potential data products.
• Preserve source versions, effective and ingestion dates, and transformation lineage. Build reproducible point-in-time datasets that reflect when information became available.
• Implement and validate time-series attributes, including changes, trends, rolling measures, and event counts, using definitions developed with business and ML teams. Account for missing periods, corrections, and changing source or matching rules.

Requirements

Required experience

• Advanced Python engineering: production-grade, reusable code for large-scale processing and automation, with strong debugging, memory management, testing, and error-handling skills.
• Advanced SQL and database development: complex joins, window functions, execution-plan analysis, data modeling, and performance optimization across large tables.
• Hands-on Spark/PySpark: building and optimizing distributed jobs, including partitioning, large joins, shuffles, data skew, and job troubleshooting.
• Production AWS experience: S3, processing services such as Glue or EMR, and secure data access patterns (IAM, encryption, least-privilege).
• Demonstrated processing experience with hundreds of millions of records, efficient storage formats such as Parquet, and practical methods for avoiding unnecessary scans and recomputation.
• Experience orchestrating multi-step data workflows with a production scheduler such as Airflow, Dagster, Step Functions, or Glue workflows, including dependency management, retries, and alerting.
• Cross-source matching, deduplication, attribute profiling, data validation, and investigation of quality issues.
• Ownership of reliable production workflows, including Git, code review, automated testing, controlled releases, monitoring, and recovery.
• Ability to interpret complex data dictionaries, translate business requirements into accurate processing logic, and communicate clearly in a remote team.
• Bachelor's degree.
• Authorized to work in the United States without current or future visa sponsorship.

Preferred experience

• SingleStore or comparable distributed SQL databases
• Apache Iceberg and historical data lake or lakehouse development
• point-in-time modeling datasets and time-series feature pipelines
• Consumer credit, property, mortgage, or direct marketing data
• CASS processing and postal data hygiene.