AI Response Quality Evaluator (Gemini)

About Gramian

Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.

About the Role

We are seeking AI Personalization Evaluators to support Gemini Personalization Evals by reviewing AI-generated responses using personalized prompts and context retrieved from connected Google applications. You will assess response quality, contextual accuracy, and personalization, identifying subtle errors, unsupported assumptions, and opportunities to improve the user experience.

This role is suited to detail-oriented individuals with strong analytical thinking, excellent written communication, and active experience using Google apps such as Gmail, Google Calendar, Google Photos, and Google Drive.

CONTRACT: Contractor

LOCATION: Remote — Australia only

TIME OVERLAP: 4 hours of overlap with PST

DURATION: Up to 16 weeks

EVALUATION PROCESS: Shortlisted candidates will receive a Job Interest Form. Selected candidates will be contacted regarding consent and onboarding requirements.

Key Responsibilities

• Evaluate AI-generated responses using personalized prompts and information retrieved from connected Google applications.
• Assess whether responses are relevant, accurate, helpful, and appropriately personalized.
• Identify incorrect personalization, unsupported assumptions, irrelevant recommendations, and inconsistencies.
• Compare multiple AI-generated responses and evaluate differences in overall user experience.
• Provide clear, detailed, and well-reasoned feedback to support AI model improvements.
• Follow project guidelines to ensure consistent and high-quality evaluations.
• Review nuanced AI responses and identify strengths, weaknesses, and subtle quality issues.
• Maintain data privacy and follow all confidentiality requirements throughout the project.

Requirements

• Currently based in Australia.
• Willing and able to connect relevant Google apps to Gemini, subject to required consent and permissions.
• Active use of Google applications, with sufficient personal data or history to support personalized retrieval evaluations.
• Strong analytical thinking and attention to detail.
• Excellent written English and ability to provide structured evaluation feedback.
• Ability to work independently in a remote environment.
• Access to a desktop or laptop with a reliable internet connection.
• Bachelor’s degree or equivalent practical experience in any field.