Sr. Data Engineer

Senior Data Engineer

Department: Technology

Reports To: Senior Manager, Data Operations

Location: Remote

FLSA Status: Exempt

People Leader: No

Travel: Less than 10%

Summary

Company Summary

Accuity partners with hospitals and health systems through a technology-enabled, physician-led model that improves clinical documentation integrity, coding accuracy, reimbursement optimization, and quality outcomes.

Job Summary

The Senior Data Engineer is a hands-on senior individual contributor within Data Operations, reporting to the Senior Manager, Data Operations. The role designs, builds, documents, and operates complex data assets across source ingestion, Bronze, Silver, and Gold on Accuity's Azure Databricks platform, including production APIs and data services.

The Senior Data Engineer translates architecture established by the principal-level Data Architect into reusable engineering patterns and production solutions. The role leads complex implementation decisions, reviews designs and code, mentors Data Engineers, and improves delivery quality and operational reliability without direct people-management responsibility.

The role leads assigned migration work from SQL Server and SSIS to Databricks and Delta Lake, including dependency analysis, validation, cutover planning, and stabilization. Data Operations owns delivery and operation across all data layers; the Data Architect retains architectural direction and the Azure team owns underlying platform capabilities.

Responsibilities

Complex Engineering and Data Modeling

•     Own complex assets from discovery and technical design through implementation, testing, documentation, deployment, and production support.

•     Build and optimize SQL, Python, and PySpark pipelines on Databricks and Delta Lake across Bronze, Silver, and Gold.

•     Implement reusable ingestion and transformation patterns covering incremental loads, change data capture, schema evolution, deduplication, late data, safe retries, and backfills.

•     Lead detailed physical and dimensional modeling for assigned solutions, including grain, keys, shared dimensions, relationships, and historical behavior.

•     Resolve implementation tradeoffs and cross-pipeline dependencies; collaborate with the Data Architect on shared definitions, significant design decisions, and exceptions.

API Design and Development

•     Independently design, build, secure, test, document, deploy, and support production APIs and data services using approved low-code, no-code, or custom development tools.

•     Define and maintain API contracts, request and response schemas, validation rules, error behavior, versioning, and backward compatibility.

•     Implement approved authentication and authorization, credential handling, least privilege, and protection of sensitive data.

•     Engineer pagination, rate limiting, retries, timeouts, and idempotency where appropriate; test performance, reliability, and consumer impact.

•     Evaluate API-based access versus persisted data delivery with the Data Architect, considering latency, source availability, replay, auditability, security, and cost.

•     Maintain interface specifications, usage examples, monitoring, and recovery procedures; coordinate changes with consumers and the Azure and application teams.

Engineering Standards and Technical Leadership

•     Translate architecture and operating standards into reference implementations, reusable components, review checklists, and practical engineering guidance.

•     Lead design and code reviews for complex or high-risk changes; improve testing, deployment automation, observability, and maintainability.

•     Mentor Data Engineers through delivery and troubleshooting while preserving their ownership of complete assets.

•     Help the manager estimate work, identify dependencies, sequence technical delivery, and assess risks and technical debt.

•     Review partner deliverables and verify that code, models, documentation, and operating knowledge transfer to internal ownership.

Data Quality and Production Reliability

•     Design automated validation and reconciliation strategies for complex assets, including edge cases, historical changes, and business-critical controls.

•     Work with business owners and analysts to establish acceptance criteria and investigate differences in meaning, timing, and transformation behavior.

•     Establish monitoring and recovery patterns for assigned pipelines and APIs; lead complex incident investigation and durable remediation under team support procedures.

•     Tune distributed processing, queries, and data services for performance, scalability, and cost; escalate infrastructure issues to the Azure team.

•     Provide technical readiness evidence for release decisions and coordinate recovery exercises and post-incident improvements.

Migration and Modernization

•     Lead technical decomposition of assigned SQL Server, stored procedure, and SSIS workloads, including undocumented rules and cross-system dependencies.

•     Design migration implementations that fit the target architecture and identify processing to simplify, consolidate, or retire.

•     Develop parallel-run, reconciliation, cutover, rollback, and stabilization plans; document intentional behavior changes for approval.

•     Coordinate execution with the manager, architect, engineers, and consumers; retire replaced components only after validation, dependency review, and approval.

•     Maintain reliable operation of remaining legacy workloads while progressing modernization.

Documentation and Knowledge Transfer

•     Create and maintain models, mappings, lineage, transformation specifications, API contracts, and dataset definitions for assigned assets.

•     Document implementation decisions, tradeoffs, dependencies, ownership, operating expectations, and known limitations.

•     Maintain runbooks for incidents, recovery, reruns, backfills, and API support; keep documentation aligned with each material change.

•     Review documentation quality and verify that other engineers can independently operate and extend delivered solutions.

Governance and Collaboration

•     Implement approved Unity Catalog organization, metadata, classification, lineage, and access patterns within delegated authority.

•     Apply Accuity security, privacy, and PHI-handling policies and applicable compliance requirements; maintain evidence for reviews and audits.

•     Partner with the Data Architect on architecture, the manager on priorities and staffing needs, and the Azure team on platform dependencies.

•     Work with analysts, business owners, and application teams on definitions and acceptance; communicate technical risks and tradeoffs clearly.

•     Escalate architecture exceptions and decisions outside delegated authority; retain hands-on responsibility for assigned delivery.

Other Duties as Assigned

•     Perform additional responsibilities as needed to support data engineering, technology, and organizational goals.

Qualifications

Education and Credentials

•     Bachelor's degree in computer science, information systems, engineering, or a related technical field preferred; equivalent professional experience may be considered.

•     Relevant Databricks, Azure, or data engineering certifications preferred.

Required Experience and Skills

•     8+ years of progressive data engineering, database development, or related experience, including technical leadership of complex production solutions.

•     Hands-on production experience with Azure Databricks, Delta Lake, Spark, and orchestration; strong SQL, Python, and PySpark proficiency.

•     Demonstrated end-to-end delivery across medallion layers and advanced practical modeling skills, including dimensional models and historical behavior.

•     Demonstrated production API design and development experience, including contracts, authentication and authorization, versioning, testing, deployment, monitoring, and support. Consuming APIs alone does not meet this requirement.

•     Ability to deliver APIs through approved low-code, no-code, or custom tools and assess when each approach is appropriate.

•     Strong understanding of JSON, XML, CSV, and standardized exchange specifications; ability to implement and validate interface mappings.

•     Experience with complex migrations, dependency analysis, reconciliation, cutover, rollback, and production troubleshooting.

•     Experience establishing reusable patterns, conducting code and design reviews, mentoring engineers, and working within architecture standards.

•     Experience with Git-based delivery, automated testing, deployment automation, data governance controls, and maintainable technical documentation.

Preferred Experience

•     Direct migration experience from SQL Server and SSIS to Databricks; familiarity with Azure Data Factory, Azure storage, Azure Functions, or API management services.

•     Experience with Unity Catalog and governing sensitive data across pipelines and APIs.

•     Healthcare and revenue cycle data experience, including HL7, FHIR, or X12 EDI and PHI handling.

•     Experience validating vendor deliverables and developing internal engineering capability.

Core Competencies

•     Technical Judgment: Resolves complex implementation tradeoffs and identifies decisions requiring architectural guidance.

•     Delivery and Reliability: Owns complete solutions and anticipates failure, recovery, consumer impact, and cost.

•     API Engineering: Delivers secure, documented interfaces with stable contracts and dependable operation.

•     Technical Leadership: Improves shared practices through reviews, reference implementations, and mentoring.

•     Documentation and Communication: Makes complex designs and risks understandable and transfers knowledge effectively.

•     Collaboration: Works across engineering, architecture, platform, and business teams without relying on reporting authority.

Additional Requirements

1) Physical Requirements: The requirements described here are representative of those that must be met by an employee to successfully perform the essential functions of this job with or without reasonable accommodations. Unless otherwise indicated, Accuity positions require interaction with people and technology while either sitting or standing. Employees must be able to communicate via phone, email, etc. and sit for extended periods of time, with or without reasonable accommodations. Physical effort and exposure to physical risk are limited to that of an office role / environment.

2) Position and Employment Statement: While this job description is intended to be an accurate reflection of the job requirements, management reserves the right to modify, add or remove duties from a job and to assign other duties as necessary and at any time. All positions at Accuity Delivery Systems, LLC, are at-will employment, and a position description is not a guarantee of a job or of job responsibilities.

While this job description is intended to be an accurate reflection of the job requirements, management reserves the right to modify, add or remove duties from a job and to assign other duties as necessary and at any time. All positions at Accuity are at-will employment, and a position description is not a guarantee of a job or of job responsibilities.