AI Evaluation Specialist, Remote - Contract

Role Title: AI Evaluation Specialist

Role Type: Contract

Location: Remote (US, CA, UK, IE, AU, NZ)

In this role, you'll apply your expertise to help train next-generation AI systems. Your work will shape how models learn, reason, and perform through high-quality, real-world input. No prior experience in AI is required — your domain knowledge is what matters.

Scope of Work

• Evaluate AI-generated outputs against detailed rubrics and defined quality standards, focusing on accuracy, relevance, and adherence to guidelines.

• Apply consistent, impartial judgment across a high volume of examples, ensuring a fair and reliable assessment process.

• Identify reasoning gaps, tool-use failures, or logic errors in AI assistant responses, providing actionable feedback for iterative improvement.

• Produce clear, concise written feedback on both strengths and areas for improvement, directly influencing model refinement and AI adoption practices.

• Participate in discussions regarding rubric interpretation and evolving quality standards, contributing to process optimization and best practices.

• Maintain meticulous documentation of evaluations and recommendations, ensuring transparency and traceability in assessment workflows.

Preferred Qualifications

• Experience in grading, quality assurance, editorial review, assessment, annotation, or similar fields demanding careful analysis and detailed feedback.

• Advanced, daily use of AI assistants (such as ChatGPT, Claude, or similar) as an essential work and productivity tool.

• Demonstrated ability to synthesize complex information and communicate findings effectively in writing.

• Background in process improvement, rubric development, or operational quality assessment in an enterprise or educational context.

• Strong critical thinking skills with a focus on consistency, integrity, and fairness in evaluations.

• Comfort working independently on large volumes of similar examples while maintaining high attention to detail.

• Collaborative mindset for sharing insights, discussing ambiguous cases, and refining evaluation criteria as models evolve.