For Employers

Softgic

Senior AI Quality Engineer (LLM Evaluation & Automation) 1754

Posted 3 months ago
Worldwide
5-10 years experience
Apply Now

Please mention DailyRemote when applying

?
Resume Match Score

See how much of this job your resume covers, and what’s missing.

Want a recruiter to go through it line by line?

Get professional review

Create a cover letter for this job

Upload your resume and we draft a letter for this exact role, tailored to what it asks for.

  • Tailored to this role
  • Based on your resume
  • Fully editable
AI Summary

Build and maintain an MVP evaluation harness including golden tasks and scorecard metrics. Integrate quality evaluations into the CI pipeline to gate releases and prevent regressions.

This is a remote position.

Owns the eval harness and quality gate from the beginning. This role replaces the old late-stage “Evals Specialist” model with a standing owner for measurable agent quality.

Key Responsibilities

• Build and maintain the MVP eval harness: golden tasks, exception tasks, scorecard metrics, and regression packs.
• Wire evals into CI so quality regressions fail builds and releases.
• Define and maintain release-gate thresholds with Product and the Tech Lead.
• Lay the path for later adversarial and drift-testing expansion without overbuilding MVP scope.


Requisitos

Must-Have Qualifications

• Experience evaluating ML, LLM, or non-deterministic systems.
• Strong test and benchmark design capability.
• Comfort working with noisy metrics, thresholds, and probabilistic behavior.
• Good scripting and automation skills.

AI-First Expectations

• Uses AI to generate candidate eval cases and failure hypotheses, but never confuses generated tests with validated quality.
• Approaches AI quality as an operating system, not a QA afterthought.

What Success Looks Like in the First 90 Days

• The first reference agent has a published scorecard and gated eval path. • Golden and exception tests run automatically. • The team can explain what “good enough to ship” means in measurable terms.

Automatically Apply to the Best Remote Jobs

Stop the endless job search. Our AI finds and applies to the best jobs for you.

Try it Now
Keep looking

Similar Jobs

See all Remote Software Development jobs →

Principal Software Engineer (m/f/d)

Full Time Germany Software Development

Support Engineer (L2)

Full Time Brazil Software Development

ML/AI Engineer - Peepal Consulting

Full Time India Software Development

Assistant Manager-Retail Credit Initiation&Control.CoE Pakistan CM - CRM - CCPL Onboarding.Credit Management-PAKCOE

Full Time Turkey Software Development

AU Service Dispatch & Operations Administrator

Full Time Philippines 2000 per month Software Development

ML/AI Engineer - Careernet

Full Time India Software Development
Apply Now

Personalize your Remote Job Search in 3 Easy Steps!

Featuring 220,192+ Jobs in Software Development

Answer easy questions

Answer easy questions

220,192+ jobs across 15+ categories

Get your best job matches

Get your best job matches

Only hand-screened, legit jobs

Find a remote job faster

Find a remote job faster

No ads, scams, or junk

“I was the first applicant for a remote marketing position that got listed on the company website the same day I applied. Had an interview within 48 hours!”

Sarah J. — Sarah J. · Marketing Manager ★★★★★ Verified