Founding AI Quality & Release Engineer

About the Role

This is a hands-on software engineering role focused on building the verification and release systems for an AI/ML product. You will develop agent-powered testing, reliable CI signals, and deployment workflows that help the team ship safer, more predictable releases.

What You'll Do

• Build and maintain agent-run test harnesses, LLM graders, and evaluation logic for AI-based systems.

• Expand regression coverage and create automated release gates that identify unsafe changes with clear evidence.

• Improve GitHub Actions workflows, container builds, CI health, and artifact publication.

• Own release pipelines and improve the reliability of staging and demo environments.

• Investigate quality gaps and incidents, then strengthen the systems that prevent them.

What We're Looking For

• Seven to fifteen years of experience in quality, release, or platform engineering.

• Recent hands-on experience applying AI-native methods to test or evaluate LLM or agentic systems, including LLM APIs, tool calling or MCP, and evaluations.

• Strong experience with GitHub Actions and GitHub-based CI/CD, including container builds and pipeline automation.

• Working knowledge of Python, TypeScript, Docker, Kubernetes, Helm, and at least one major cloud or data platform such as AWS, Azure, Databricks, or Snowflake.

• Experience taking independent ownership of systems, ideally in a fast-moving startup or while modernizing established QA or DevOps practices.

• Authorization to work in the US or Canada without visa sponsorship.

Compensation & Benefits

Base salary is $170,000 to $200,000 USD annually. Visa sponsorship is not available.

Location

Remote, with New York, New York as the primary location.