Please mention DailyRemote when applying
See how much of this job your resume covers, and what’s missing.
Want a recruiter to go through it line by line?
Get professional reviewUpload your resume and we draft a letter for this exact role, tailored to what it asks for.
Build and maintain automated AI-powered test harnesses, LLM graders, and evaluation logic to ensure reliable release systems. Develop automated release gates and manage CI/CD pipelines to improve the trustworthiness of test signals and prevent unsafe releases.
Join the engineering team as its first dedicated quality-infrastructure hire. This is a hands-on software engineering role focused on building AI-powered verification and reliable release systems, so teams can trust test results, ship safely, and make releases more predictable. The work is centered on automated systems, not manual test execution.
Build and maintain agent-powered test harnesses, LLM graders, evaluation logic, and regression coverage for AI systems.
Improve the trustworthiness of test signals and close gaps that allow defects to reach customers.
Develop automated release gates that make readiness visible and prevent unsafe releases based on evidence.
Own CI/CD pipelines, container builds, artifact publication, and release automation using GitHub Actions.
Improve the reliability of pre-production, staging, and demo environments.
Participate in business-hours escalations and make issues easier to diagnose, without formal after-hours production on-call.
Seven to 15 years of experience in quality engineering, release engineering, or platform engineering.
Recent hands-on quality engineering experience applying AI-native testing or evaluation to LLM or agentic systems.
Practical experience with LLM APIs, tool calling or MCP, and evaluation methods.
Strong familiarity with GitHub Actions and GitHub-based CI/CD, including container builds and pipeline automation.
Experience with Python, TypeScript, Docker, Kubernetes, and Helm, plus familiarity with a major cloud or data platform such as AWS, Azure, Snowflake, or Databricks.
Background in an early-stage, fast-moving startup or experience modernizing legacy QA or DevOps practices, with the ability to own systems independently.
Authorization to work in the United States or Canada without visa sponsorship.
Annual salary range: $170,000 to $200,000 USD. Visa sponsorship is not available.
Remote, with the primary location listed as New York, New York, United States. Candidates must be authorized to work in the United States or Canada.
Stop the endless job search. Our AI finds and applies to the best jobs for you.
Featuring 215,628+ Jobs in Software Development
Answer easy questions
215,628+ jobs across 15+ categories
Get your best job matches
Only hand-screened, legit jobs
Find a remote job faster
No ads, scams, or junk
“I was the first applicant for a remote marketing position that got listed on the company website the same day I applied. Had an interview within 48 hours!”
— Sarah J.
·
Marketing Manager
★★★★★
Verified
