Senior Automation Specialist, AI Operations
Please mention DailyRemote when applying
AI Summary
The Senior Automation Specialist will own evaluation and dataset workstreams to improve AI system performance. They will design measurement frameworks, build evaluation systems, and create automations to streamline operational workflows.
About the Role:
At Fetch, we’re building AI and automation systems that make our work smarter, faster, and more scalable. The AI Operations team ensures our models and workflows perform with quality, reliability, and measurable impact.
As a Senior Automation Specialist, you’ll own evaluation and dataset workstreams that improve AI system performance. You’ll define what “good” looks like, design how we measure it, and build the datasets, evaluations, and automations that turn insights into action.
This role is ideal for someone who is self-directed, biases toward action and rapid iteration, and takes initiative to turn ambiguous problems into structured, scalable solutions.
This is a full-time role that can be held from one of our US offices or remotely in the United States.
Role Responsibilities:
- Own evaluation & datasets: Define evaluation approaches, design gold datasets (GDS), and ensure coverage of real-world scenarios and edge cases
- Build evaluation systems: Develop manual and automated evals, including LLM-as-judge patterns, to measure model quality and performance
- Translate ambiguity into structure: Turn open-ended questions into clear evaluation frameworks and execution plans
- Build automations: Create automations that improve workflows, including dataset creation, evaluation pipelines, and lightweight operational processes
- Measure and iterate: Define and track performance metrics; refine datasets, evaluations, and workflows based on results
- Drive execution forward: Operate with urgency and ownership; identify next steps, unblock progress, and move work forward with minimal oversight
- Collaborate cross-functionally: Partner with other Automation Specialists, engineering, cross-functional stakeholders, and project leads to ensure high-quality, timely project deliverables
- Improve systems: Identify gaps and implement scalable improvements to evaluation and data workflows
Minimum Requirements:
- 3+ years of experience designing or working with evaluation frameworks, datasets, or quality measurement systems
- Experience building or managing datasets (labeling, QA, iteration)
- Ability to independently drive tasks from problem definition to execution
- Hands-on experience with AI tools, LLM workflows, or automation platforms
- Experience in defining and tracking model performance
- Basic scripting or data skills (SQL, Python, etc.)
Preferred Requirements:
- Experience with LLM-as-judge or model evaluation techniques
- Familiarity with prompt evaluation or benchmarking approaches
- Experience productionizing evaluation workflows with engineering teams
Compensation:
At Fetch, we offer competitive compensation packages including base, equity, and benefits to the exceptional folks we hire. The base salary range for this position is [$75,000 - $85,000]. Discover our benefits and how our employees live rewarded at https://fetch.com/careers.
Automatically Apply to the Best Remote Jobs
Stop the endless job search. Our AI finds and applies to the best jobs for you.
