Guide merchants through the setup and integration of online ordering platforms with their websites and Google profiles. Handle technical troubleshooting, POS integrations, and brand customization to ensure a seamless digital commerce experience.
Provide native-level Haitian Creole linguistic vetting and quality assurance for AI data outputs. Develop educational resources and feedback documentation to improve AI alignment and cultural accuracy.
Record high-fidelity, professional-grade audio of humming exercises to train AI audio models. Ensure strict adherence to tonal and rhythmic prompts while maintaining technical audio quality.
The specialist will challenge and train advanced AI language models by analyzing Russian linguistic structures and documenting failure modes. Responsibilities include verifying factual accuracy, capturing error traces, and improving prompt engineering metrics.
The specialist will challenge and train advanced AI language models by analyzing Arabic grammar, syntax, and cultural context. Responsibilities include verifying factual accuracy, documenting error traces, and improving prompt engineering metrics.
Provide native-level Azerbaijani linguistic QA, data annotation, and validation for AI outputs to ensure naturalness and cultural accuracy. Develop educational resources and feedback documentation to improve AI alignment with campaign expectations.
Provide native-level Arabic linguistic QA, data annotation, and validation to ensure AI outputs are natural and culturally accurate. Develop educational resources and feedback documentation to improve AI alignment with project expectations.
Provide native-level Estonian linguistic QA, data annotation, and validation for AI outputs to ensure naturalness and cultural accuracy. Develop educational resources and feedback documentation to improve AI alignment with campaign expectations.
Evaluate AI-driven commerce tools and digital merchant features to provide structured feedback on usability and accuracy. Audit local search representations and business summaries to ensure empirical accuracy against real-world operations.
Provide native-level Māori language vetting, QA, and data annotation to ensure AI outputs are natural and culturally accurate. Develop educational resources and feedback documentation to improve AI alignment with campaign expectations.
Provide native-level Taiwanese linguistic vetting and quality assurance for AI data projects. Develop educational resources and feedback documentation to improve AI alignment and cultural accuracy.
Generate collision-accurate 3D meshes from 2D images and write procedural generation scripts using Python or JavaScript. Deliver structured programmatic outputs and maintain detailed trajectory logs to train AI systems on geometric construction.
Create and execute role-play scenarios to evaluate and benchmark agentic audio models across various customer support domains. Contribute to the development of representative datasets and assess model performance based on qualitative and quantitative metrics.
The specialist will audit, rewrite, and refine AI-generated content to ensure stylistic consistency and technical accuracy. Responsibilities include correcting LaTeX mathematical expressions and fixing markdown formatting errors.
You will converse with AI models to verify factual accuracy and logical soundness while documenting failure modes. Additionally, you will suggest improvements to prompt engineering and evaluation metrics to enhance model reasoning.
The specialist will autonomously design complex evaluation frameworks and generate high-quality prompts to train AI models. They are responsible for producing reliable benchmark data and ensuring all tasks meet strict search quality and indexing standards.
You will converse with AI models to verify factual accuracy, logical soundness, and linguistic quality. You will also document failure modes and suggest improvements to prompt engineering and evaluation metrics.
You will converse with AI models to test linguistic accuracy, document failure modes, and suggest improvements to prompt engineering. Your work involves verifying factual soundness and capturing error traces to harden model reasoning.
You will converse with advanced language models to verify factual accuracy, logical soundness, and linguistic precision. Additionally, you will document failure modes and suggest improvements for prompt engineering and evaluation metrics.
You will converse with and challenge advanced language models to identify errors in grammar, syntax, and logic. Additionally, you will document failure modes and suggest improvements to prompt engineering and evaluation metrics.
The specialist will autonomously design complex evaluation frameworks and generate high-quality prompts to train AI models. They are responsible for creating structured rubrics and ensuring all data meets strict industry and regulatory standards.
The specialist will autonomously design complex evaluation frameworks and generate high-quality prompts to train AI models in scientific and technical domains. They are responsible for creating objective scoring rubrics and performing rigorous fact-checking to ensure data reliability.
The specialist will autonomously design complex evaluation frameworks and generate high-quality prompts to train AI models in healthcare and social assistance tasks. They are responsible for developing objective scoring rubrics and ensuring all generated data adheres to strict clinical and regulatory standards.
The Manufacturing Specialist will design complex evaluation frameworks and generate high-quality prompts to train AI models on industrial workflows. They are responsible for creating objective scoring rubrics and ensuring all data reflects real-world safety and production standards.
The specialist will design complex evaluation frameworks and generate high-quality prompts to train AI models on public administration and policy tasks. They are also responsible for developing objective scoring rubrics and performing rigorous fact-checking to ensure data reliability.
The specialist will autonomously design complex evaluation frameworks and generate high-quality prompts to train AI models in retail strategies. They are responsible for developing clear scoring rubrics and ensuring all training data adheres to real-world retail market standards.
The Information Specialist will autonomously design complex evaluation frameworks and generate high-quality training data for AI models. Responsibilities include creating realistic prompts, developing objective scoring rubrics, and performing quality assurance on data architectures.
The specialist will design complex evaluation frameworks and generate high-quality prompts to train AI models on wholesale and B2B distribution tasks. They are also responsible for developing objective scoring rubrics and performing quality assurance to ensure data accuracy against real-world market standards.
The specialist will design complex evaluation frameworks and generate high-quality prompts to train AI models in real estate and leasing tasks. They are responsible for developing clear scoring rubrics and ensuring all data meets professional industry standards through rigorous fact-checking.
You will evaluate model-generated Java code for correctness, reasoning, and production quality while identifying logic failures. Additionally, you will document these failure patterns and assess the suitability of Java problems for AI training purposes.
You will converse with AI models on real-world coding scenarios to verify factual accuracy and logical soundness. Additionally, you will document failure modes, capture error traces, and suggest improvements to prompt engineering and evaluation metrics.
You will converse with AI models to test language capabilities, verify factual accuracy, and document failure modes. Additionally, you will suggest improvements to prompt engineering and evaluation metrics to enhance model reasoning.
You will converse with advanced language models to test their linguistic accuracy, reasoning, and factual soundness. You are responsible for documenting failure modes and suggesting improvements to prompt engineering and evaluation metrics.
You will converse with and challenge advanced language models to identify failure modes and document errors. Additionally, you will verify factual accuracy and suggest improvements to prompt engineering and evaluation metrics.
You will converse with AI models to test language capabilities, document error traces, and verify factual accuracy. Additionally, you will suggest improvements to prompt engineering and evaluation metrics to enhance model performance.
You will converse with and challenge advanced language models to identify failures in Bulgarian grammar, syntax, and nuance. Additionally, you will verify factual accuracy, document error traces, and suggest improvements to prompt engineering and evaluation metrics.
You will converse with advanced language models to evaluate factual accuracy, logical soundness, and linguistic nuances. You are responsible for documenting failure modes and suggesting improvements to prompt engineering and evaluation metrics.
You will evaluate and harden large language models by conducting adversarial testing, verifying factual accuracy, and documenting failure modes. Additionally, you will design test plans, build rubrics, and suggest improvements to prompt engineering and model guardrails.
You will converse with advanced language models to verify factual accuracy, logical soundness, and linguistic precision. You are responsible for documenting failure modes and suggesting improvements to prompt engineering and evaluation metrics.
The role involves performing linguistic quality assurance, annotating AI data, and validating outputs for grammatical accuracy and cultural nuance. Additionally, the partner will develop educational resources to align AI task outputs with project expectations.
The specialist will autonomously guide merchants through the setup and integration of online ordering platforms, including website customization and POS system configuration. They are also responsible for leading high-volume, video-based consultation meetings to address merchant needs and resolve technical barriers.
The role involves performing linguistic quality assurance, annotating AI data, and validating outputs for grammatical accuracy and cultural nuance. Additionally, the partner will develop educational resources to improve AI alignment and provide ongoing linguistic consultation.
The role involves performing linguistic quality assurance, annotating AI data, and validating outputs for naturalness and cultural accuracy. Additionally, the partner will develop educational resources and feedback documentation to improve AI model alignment.
The role involves performing linguistic quality assurance, annotating AI data, and validating outputs for grammatical and cultural accuracy. Additionally, the partner will develop educational resources to improve AI alignment with campaign expectations.
You will converse with and challenge advanced language models to improve their reasoning and linguistic accuracy. This involves documenting failure modes, verifying factual soundness, and suggesting improvements to prompt engineering and evaluation metrics.
You will converse with AI models in Egyptian Arabic to verify linguistic accuracy, cultural relevance, and semantic precision. Additionally, you will document error traces and suggest improvements to prompt engineering and evaluation metrics.
You will converse with AI models to verify factual accuracy, logical soundness, and linguistic nuances. Additionally, you will document failure modes and suggest improvements to prompt engineering and evaluation metrics.
The Product Matching Specialist will evaluate digital content, accurately match new products to existing ones, and create new product entries as needed. They will also conduct quality assurance auditing and provide feedback on tagging logic and taxonomy.
The primary role involves engaging AI models with realistic small business scenarios and operational challenges, evaluating outputs related to key business functions like budgeting and marketing strategy. Responsibilities also include assessing practicality, capturing error traces, and providing structured feedback to enhance the model's real-world applicability for Very Small Businesses (VSBs).
Participants will engage the AI model using realistic small business scenarios and operational challenges, evaluating outputs related to budgeting, marketing, hiring, and compliance to ensure practicality for Very Small Businesses. The role involves capturing error traces and providing structured feedback to enhance prompts and real-world applicability of the AI responses.
The specialist will converse with advanced language models on various language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Daily tasks also involve suggesting improvements to prompt engineering and evaluation metrics to harden model reasoning.
The specialist will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Daily tasks also involve suggesting improvements to prompt engineering and evaluation metrics.
Responsibilities involve interacting directly with AI systems, reviewing model outputs, and following clear task guidelines to contribute to the creation of high-quality training data. This includes activities like evaluating responses, identifying inconsistencies, and providing feedback to help models learn more effectively.
The specialist will converse with advanced language models on various language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Daily tasks also involve suggesting improvements to prompt engineering and evaluation metrics to harden model reasoning.
The specialist will converse with advanced language models on various language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Daily tasks include suggesting improvements to prompt engineering and evaluation metrics based on model failures.
Annotators will review social media videos, posts, and other media to assess whether specific criteria are met, and evaluate model outputs across various categories against specified criteria. Daily tasks involve conversing with the model on language scenarios, verifying factual accuracy, capturing error traces, and suggesting improvements to prompt engineering and evaluation metrics.
Specialists will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces related to Tsugaru language nuances. They will also suggest improvements to prompt engineering and evaluation metrics to harden model reasoning.
Specialists will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces to harden model reasoning. Daily tasks involve documenting failures related to Carioca grammar, syntax, and stylistic variation.
The specialist will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces related to Alentejano dialect nuances. They will also suggest improvements to prompt engineering and evaluation metrics based on their findings.
The specialist will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Daily tasks also involve suggesting improvements to prompt engineering and evaluation metrics based on documented model failures.
Specialists will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces related to Nortenho dialect nuances. They will also suggest improvements to prompt engineering and evaluation metrics to harden model reasoning.
Specialists will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces related to Wu language nuances. They will also suggest improvements to prompt engineering and evaluation metrics to enhance model reasoning.
Specialists will converse with advanced language models on various language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces to harden model reasoning. Daily tasks include suggesting improvements to prompt engineering and evaluation metrics based on model failures.
Specialists will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces related to Xiang language nuances. Daily tasks include suggesting improvements to prompt engineering and evaluation metrics based on model failures.
The specialist will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces to harden model reasoning. Daily tasks involve documenting model failures related to Bhojpuri grammar, syntax, and semantics.
Specialists will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces related to the Cordobés dialect. They will also suggest improvements to prompt engineering and evaluation metrics to harden model reasoning.
Specialists will converse with advanced language models on Litoraleño language scenarios, verifying factual accuracy and logical soundness while capturing reproducible error traces. They will also suggest improvements to prompt engineering and evaluation metrics to harden model reasoning.
Specialists will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces related to Norteño language nuances. They will also suggest improvements to prompt engineering and evaluation metrics to harden model reasoning.
Specialists will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces to harden model reasoning. Daily tasks involve documenting failures related to Yucateco grammar, syntax, and stylistic variation.
The specialist will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Daily tasks also involve suggesting improvements to prompt engineering and evaluation metrics based on model failures.
Specialists will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces to harden model reasoning. Daily tasks involve documenting failures related to Bedawi grammar, syntax, and stylistic variation.
The specialist will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces related to Khaleeji language nuances. Daily tasks also involve suggesting improvements to prompt engineering and evaluation metrics to harden model reasoning.
Specialists will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces to harden model reasoning. They will also suggest improvements to prompt engineering and evaluation metrics based on documented failure modes.
Specialists will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces related to Kansai dialect nuances. They will also suggest improvements to prompt engineering and evaluation metrics to harden model reasoning.
Consultants will define the 'Gold Standard' rubrics for evaluating AI outputs by reviewing quality criteria and determining if AI logic meets professional scrutiny in their field. Key deliverables include curating 'Golden Tasks' (benchmark examples) and providing expert gap analysis on model failures.
Specialists will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces to harden model reasoning. Daily tasks involve documenting failures related to Sunda grammar, syntax, and semantics to improve prompt engineering and evaluation metrics.
The specialist will converse with advanced language models on various language scenarios, verifying factual accuracy and logical soundness of responses. Daily tasks include capturing reproducible error traces and suggesting improvements to prompt engineering and evaluation metrics.
The specialist will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Daily tasks also involve suggesting improvements to prompt engineering and evaluation metrics based on model failures.
The Implementation Operations Agent will be crucial to the Implementation team by operationally setting up the client's plan platform. This role ensures the platform is configured correctly and running at full speed for the client.
The specialist will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Daily tasks involve challenging models on German linguistic aspects like verb conjugation and case systems to harden model reasoning.
The specialist will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. They will also suggest improvements to prompt engineering and evaluation metrics for the AI.
The specialist will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces to harden model reasoning. Daily tasks include suggesting improvements to prompt engineering and evaluation metrics based on documented failure modes.
The Operations Agent will be responsible for accurately and efficiently processing participant requests for distributions and loans from retirement plans while adhering to all procedures and regulatory requirements. This includes handling a high volume of time-sensitive tickets, making outbound calls, sending professional emails, and accurately documenting all interactions in tracking systems like Jira.
Contractors will engage AI models on complex educational scenarios, validating pedagogical concepts and evaluating AI outputs for clarity and correctness. This involves analyzing reasoning errors, documenting logic gaps, and providing structured, rewritten explanations to improve how the model reasons and teaches.
Contractors will challenge AI models using realistic financial scenarios involving regulatory compliance, illegal actions, and complex conversational testing to evaluate system guardrails. Responsibilities include validating regulatory understanding, scoring factual accuracy against ground truth, analyzing safety bypasses, and documenting potential regulatory loopholes.
The role involves recording scripted material in Japanese, assessing AI-generated speech outputs for linguistic accuracy and naturalness, and annotating any errors found. Additionally, the contractor will collaborate with the team to refine prompts, evaluation methods, and voice design guidelines.
The role involves recording scripted material in Portuguese, assessing AI-generated speech outputs for linguistic accuracy and naturalness, and annotating any errors found. Additionally, the voice actor will collaborate with the team to refine prompts, evaluation methods, and voice design guidelines.
The role involves recording scripted material in Finnish, assessing AI-generated speech outputs for accuracy and naturalness, and annotating any errors found. Additionally, the contractor will collaborate with the team to refine prompts, evaluation methods, and voice design guidelines for Finnish voice models.
The role involves recording scripted material in Welsh, assessing AI-generated speech outputs for accuracy and naturalness, and annotating any errors found. Responsibilities also include collaborating with the team to refine evaluation methods, prompts, and voice design guidelines.
The primary duties involve recording scripted material in Russian, assessing AI-generated speech outputs for linguistic accuracy and naturalness, and annotating errors found in the output. Additionally, the role requires collaboration with the team to refine prompts, evaluation methods, and voice design guidelines.
The role involves recording scripted material in Arabic, assessing AI-generated speech outputs for accuracy and naturalness, and annotating any errors found. Additionally, the contractor will collaborate with the team to refine prompts, evaluation methods, and voice design guidelines for Arabic voice models.
The primary duties involve recording scripted material in Icelandic, assessing AI-generated speech outputs for linguistic accuracy and naturalness, and annotating any errors found. Additionally, the role requires collaboration with the team to refine prompts, evaluation methods, and voice design guidelines.
The role involves recording scripted material in Romanian, assessing AI-generated speech outputs for linguistic accuracy and naturalness, and annotating any errors found. Additionally, the contractor will collaborate with the team to refine prompts, evaluation methods, and voice design guidelines for AI models.
The primary duties involve recording scripted material in Hindi, assessing AI-generated speech outputs for linguistic accuracy and naturalness, and annotating errors found in the outputs. The role also requires collaborating with the team to refine prompts, evaluation methods, and voice design guidelines for Hindi voice models.
The primary duties involve recording scripted material in Czech, assessing AI-generated speech outputs for linguistic accuracy and naturalness, and annotating any errors found. Additionally, the role requires collaboration with the team to refine prompts, evaluation methods, and voice design guidelines.
The primary duties involve recording scripted material in Estonian, assessing AI-generated speech outputs for linguistic accuracy and naturalness, and annotating errors found in the data. Additionally, the role requires collaboration with the team to refine prompts, evaluation methods, and voice design guidelines.
The primary duties involve recording scripted material in Maltese, assessing AI-generated speech outputs for accuracy and naturalness, and annotating any errors found. Additionally, the role requires collaboration with the team to refine evaluation methods, prompts, and voice design guidelines.
The role involves recording scripted material in Latvian, assessing AI-generated outputs for linguistic accuracy and naturalness, and annotating errors. Additionally, the contractor will collaborate with the team to refine prompts, evaluation methods, and voice design guidelines.
The primary duties involve recording scripted material in Lithuanian, assessing AI-generated speech outputs for accuracy and naturalness, and annotating any errors found. Additionally, the role requires collaboration with the team to refine prompts, evaluation methods, and voice design guidelines for AI models.
The role involves recording scripted material in Slovene, assessing AI-generated speech outputs for linguistic accuracy and naturalness, and annotating errors to strengthen Slovene voice models. Additionally, the contractor will collaborate with the team to refine prompts, evaluation methods, and voice design guidelines.
This role involves interacting directly with an AI-generated world model to navigate virtual environments, complete defined objectives, and evaluate the system's responsiveness to player inputs and in-game scenarios. Responsibilities include assessing world coherence, identifying inconsistencies, and providing clear, structured feedback on performance gaps and environmental realism.
The primary duties involve recording scripted material, assessing AI-generated speech outputs for accuracy and naturalness, and annotating any errors found. Additionally, the role requires collaboration with the team to refine prompts, evaluation methods, and voice design guidelines.
Responsibilities include creating and executing role-play scenarios simulating customer service interactions across domains like travel, finance, and telecommunications. The specialist will also contribute to developing diverse datasets and evaluating model performance against qualitative and quantitative metrics.
The specialist will create and execute role-play scenarios simulating customer service interactions across domains like flight bookings, finance, and telecommunications to evaluate advanced agentic audio models. Responsibilities also include contributing to diverse datasets and assessing model performance against qualitative and quantitative metrics.
The specialist will create and execute role-play scenarios simulating customer service interactions across domains like travel and finance to assess advanced agentic audio models. Responsibilities also include contributing to diverse datasets and evaluating model performance against qualitative and quantitative metrics.
The specialist will review and annotate Amharic content, assess AI-generated outputs for accuracy and fluency, and identify/document error patterns. They will also collaborate with the team to refine prompts, evaluation methods, and linguistic guidelines.
Specialists will engage with advanced language models in Albanian and English to evaluate generated text for linguistic accuracy, annotate errors, provide corrected versions, and document patterns in model performance. This includes assessing grammar, fluency, tone, and handling region-specific variations and cultural context.
The specialist will engage advanced language models daily with investment scenarios, analytical questions, and market-based reasoning tasks to verify factual correctness and financial logic. Responsibilities include assessing the validity of investment reasoning, capturing error traces, and providing structured feedback to enhance model prompts and analytical depth.
The specialist will engage advanced language models with financial institution scenarios, regulatory questions, and analytical exercises, verifying factual correctness and institutional logic. Responsibilities include assessing risk-related reasoning, capturing error traces, and providing structured feedback to enhance model prompts and analytical depth.
Engage advanced language models with corporate finance and market-based scenarios, verifying financial accuracy, assessing valuation validity, capturing error traces, and providing structured feedback to enhance model reasoning and analytical precision. Identify instances where models oversimplify complex transactions or misinterpret regulatory or market signals.
Engage advanced language models daily with business scenarios, analytical questions, and strategy exercises to evaluate logical consistency and real-world applicability. Capture reproducible error traces and provide structured feedback to enhance model reasoning, prompt quality, and evaluation frameworks.
Specialists will engage advanced language models with policy scenarios, verify factual and institutional correctness, assess logical consistency and bias risks, and capture reproducible error traces. They will also provide structured feedback to enhance model prompts, evaluation standards, and analytical depth regarding political and governance topics.
Engage advanced language models with religious or historical scenarios, analytical questions, and cultural context tasks, evaluating accuracy, neutrality, and logical consistency. Capture reproducible error traces and provide structured feedback to enhance prompt quality, evaluation frameworks, and model reasoning.
Specialists will engage advanced language models with literary passages, analytical questions, and interpretation tasks, verifying factual accuracy and evaluating the depth and validity of literary analysis provided by the AI.
The specialist will converse with advanced language models on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. They will also suggest improvements to prompt engineering and evaluation metrics to harden model reasoning.
The specialist will review roadway images and inspection records to validate AI-generated outputs related to pavement distress and condition assessments. They will also annotate pavement features and provide feedback on model accuracy and classification.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and document error traces. Your role will also involve suggesting improvements to prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and document failures to improve AI models. Your expertise will help shape the training data for the next generation of AI.
You will engage with advanced language models on various language scenarios, verifying their accuracy and documenting errors. Your role will also involve suggesting improvements to prompt engineering and evaluation metrics.
The primary responsibility involves applying civil or transportation engineering expertise to review and evaluate AI-generated outputs concerning roadway conditions, inspection workflows, and infrastructure assessment. This includes validating pavement condition classifications, checking data interpretations, and providing structured feedback to improve next-generation AI models.
You will challenge advanced language models on various German language topics and document failures to improve model reasoning. Daily tasks include conversing with the model, verifying accuracy, and suggesting improvements.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to our prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to prompt engineering and evaluation metrics.
You will converse with the model on real-world coding scenarios and theoretical computer science questions, verifying factual accuracy and logical soundness. Additionally, you will capture reproducible error traces and suggest improvements to prompt engineering and evaluation metrics.
You will engage with AI across various topics, review its responses for quality, and document errors or gaps. Collaborating with the team, you will refine prompts, tasks, and evaluation methods to improve model performance.
You will evaluate AI-generated outputs, identify analytical weaknesses, and design metrics and test cases to advance model understanding and predictive accuracy. Daily tasks include reviewing geospatial scenarios, validating model outputs, and collaborating with the team to improve data-driven evaluations.
You will evaluate AI-generated outputs, identify analytical weaknesses, and design metrics and test cases that advance model understanding and predictive accuracy. Additionally, you will review geospatial scenarios, validate model outputs, and collaborate with the team to improve data-driven evaluations.
You will converse with the model on real-world finance scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to our prompt engineering and evaluation metrics.
You will interact directly with AI systems, review model outputs, and contribute to the creation of high-quality training data. Tasks include evaluating responses, identifying inconsistencies, and providing feedback.
You will converse with the model on clinical scenarios and theoretical medical questions, verifying factual accuracy and logical soundness. Additionally, you will capture reproducible error traces and suggest improvements to prompt engineering and evaluation metrics.
You will converse with the model on document preparation scenarios and theoretical LaTeX questions, verifying formatting accuracy and logical consistency. Additionally, you will capture reproducible error traces and suggest improvements to prompt engineering and evaluation metrics.
United States$6 - $65 per hour5-10 yrs expTeaching
You will converse with the model on classroom problems and theoretical mathematics questions, verifying factual accuracy and logical soundness. Additionally, you will capture reproducible error traces and suggest improvements to prompt engineering and evaluation metrics.
You will record scripted material, assess AI-generated outputs for linguistic accuracy and naturalness, and annotate errors. Collaboration with the team to refine prompts, evaluation methods, and voice design guidelines is also expected.
You will converse with the model on software engineering tasks and technical scenarios in German, verifying logical accuracy and coding fluency. Additionally, you will assess the naturalness and correctness of German language usage and suggest improvements to prompt engineering and evaluation metrics.
You will apply your experience as a detective or police officer to interact with AI models on law enforcement-related scenarios. This includes reviewing and evaluating model responses and providing written feedback on their accuracy and realism.
Prepare, clean, and optimize spoken-word audio for post-production delivery. Diagnose, flag, and edit audio to meet client specifications while collaborating with technical teams.
You will engage with AI across various topics, review its responses for quality, and document errors or gaps. Collaborating with the team, you will refine prompts, tasks, and evaluation methods to improve model performance.
You will review and discuss garment quality inspection scenarios with the AI model and verify whether the model's reasoning aligns with real factory QC practices. Additionally, you will evaluate defect classification and suggest improvements to training prompts.
You will review and discuss garment quality inspection scenarios with the AI model and verify whether its reasoning aligns with real factory QC practices. Additionally, you will evaluate defect classification and suggest improvements to training prompts.
You will review and discuss garment quality inspection scenarios with the AI model, verifying its reasoning against real factory QC practices. Additionally, you will evaluate defect classification and suggest improvements to training prompts.
You will review and discuss garment quality inspection scenarios with the AI model and verify whether model reasoning aligns with real factory QC practices. Additionally, you will evaluate defect classification and severity assessment, analyze inspection workflows, and suggest improvements to training prompts.
You will review and discuss garment quality inspection scenarios with the AI model and verify whether the model's reasoning aligns with real factory QC practices. Additionally, you will evaluate defect classification, analyze inspection workflows, and suggest improvements to training prompts.
You will review and discuss garment quality inspection scenarios with the AI model and verify whether its reasoning aligns with real factory QC practices. Additionally, you will evaluate defect classification, analyze inspection workflows, and suggest improvements to training prompts.
You will review and discuss garment quality inspection scenarios with the AI model and verify whether model reasoning aligns with real factory QC practices. Additionally, you will evaluate defect classification and suggest improvements to training prompts.
You will converse with the model on editing scenarios and theoretical audio engineering questions, verifying technical accuracy and creative logic. Additionally, you will capture reproducible error traces and suggest improvements to prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and document error traces. Additionally, you will suggest improvements to prompt engineering and evaluation metrics.
You will review and discuss garment quality inspection scenarios with the AI model and verify whether its reasoning aligns with real factory QC practices. Additionally, you will evaluate defect classification, analyze inspection workflows, and suggest improvements to training prompts.
You will review and discuss garment quality inspection scenarios with the AI model and verify whether its reasoning aligns with real factory QC practices. Additionally, you will evaluate defect classification, analyze inspection workflows, and suggest improvements to training prompts.
You will record scripted material and assess AI-generated outputs for linguistic accuracy and naturalness. Additionally, you will annotate errors and collaborate with the team to refine prompts and voice design guidelines.
You will review and discuss garment quality inspection scenarios with the AI model and verify whether the model's reasoning aligns with real factory QC practices. Additionally, you will evaluate defect classification and suggest improvements to training prompts.
You will record scripted material and assess AI-generated outputs for linguistic accuracy and naturalness. Additionally, you will annotate errors and collaborate with the team to refine prompts and voice design guidelines.
You will review and discuss garment quality inspection scenarios with the AI model and verify whether its reasoning aligns with real factory QC practices. Additionally, you will evaluate defect classification and suggest improvements to training prompts.
You will review and discuss garment quality inspection scenarios with the AI model and verify whether its reasoning aligns with real factory QC practices. Additionally, you will evaluate defect classification and suggest improvements to training prompts.
You will converse with the AI model on language scenarios, verify factual accuracy, and capture reproducible error traces. Additionally, you will suggest improvements to prompt engineering and evaluation metrics.
You will converse with the model on editing scenarios and theoretical audio engineering questions, verifying technical accuracy and creative logic. Additionally, you will capture reproducible error traces and suggest improvements to prompt engineering and evaluation metrics.
The role involves supporting Self Service Plan Sponsors during their onboarding process by providing timely guidance and addressing inquiries. The Specialist monitors onboarding progress, identifies blockers, and communicates effectively with stakeholders.
The role involves supporting Self Service Plan Sponsors during their onboarding process by providing timely guidance and addressing inquiries. The Specialist monitors onboarding progress and helps clients navigate the platform effectively.
The role involves supporting Self Service Plan Sponsors during onboarding by providing guidance and responding to inquiries. The Specialist monitors progress, identifies blockers, and communicates with stakeholders to ensure successful onboarding.
The role involves supporting Self Service Plan Sponsors during their onboarding process by providing timely guidance and addressing inquiries. The Specialist monitors onboarding progress, identifies blockers, and communicates effectively with stakeholders.
The role involves supporting Self Service Plan Sponsors during their onboarding process by providing guidance and responding to inquiries. The Specialist monitors onboarding progress and communicates effectively with stakeholders to ensure a smooth onboarding experience.
You will converse with the model on editing scenarios and theoretical audio engineering questions, verifying technical accuracy and creative logic. Additionally, you will capture reproducible error traces and suggest improvements to prompt engineering and evaluation metrics.
You will converse with the model on real-world ASL communication scenarios and verify linguistic accuracy and cultural appropriateness. Additionally, you will document failures and suggest improvements to prompt engineering and evaluation metrics.
You will converse with the model on clinical scenarios and nursing judgment questions, verifying factual accuracy and logical soundness. Additionally, you will document errors and suggest improvements to enhance the model's performance.
You will converse with the model on software engineering tasks and technical scenarios using Python, verifying logical accuracy and coding fluency. Additionally, you will assess code quality and clarity, capture reproducible error traces, and suggest improvements to prompt engineering and evaluation metrics.
You will converse with the model on infrastructure and platform engineering tasks, verifying architectural soundness and assessing code quality. Additionally, you will analyze performance bottlenecks and suggest improvements to strengthen model reasoning.
You will engage the model on Java-centric infrastructure tasks and verify architectural decisions related to performance metrics. Additionally, you will assess testing strategies and analyze deployment risks.
You will converse with the model on software engineering tasks and technical scenarios using TypeScript, verifying logical accuracy and coding fluency. Additionally, you will assess code quality and clarity, capture reproducible error traces, and suggest improvements to prompt engineering and evaluation metrics.
You will converse with the model on software engineering tasks and technical scenarios using Rust, verifying logical accuracy and coding fluency. Additionally, you will assess code quality and clarity, capture reproducible error traces, and suggest improvements to prompt engineering and evaluation metrics.
You will converse with the model on software engineering tasks and technical scenarios using PHP, verifying logical accuracy and coding fluency. Additionally, you will assess code quality and clarity, capture reproducible error traces, and suggest improvements to prompt engineering and evaluation metrics.
You will evaluate diverse drone images and apply strict annotation standards designed to optimize downstream model performance. This includes isolating the core structure of a drone while excluding peripheral or transient elements.
You will engage with advanced theoretical problems, construct and evaluate proofs, and analyze reasoning steps for validity. Additionally, you will collaborate with the team to refine prompts and evaluation frameworks.
You will analyze AI-generated graphics, evaluate composition and visual reasoning, and provide expert feedback on design accuracy and adherence to principles. Collaborating with the team, you will refine prompts and design quality guidelines.
You will converse with the model on real-world BSL communication scenarios and verify linguistic accuracy and cultural appropriateness. Additionally, you will document failures and suggest improvements to enhance the model's performance.
You will converse with the model on real-world BSL communication scenarios and verify linguistic accuracy and cultural appropriateness. Your role includes documenting failures and suggesting improvements to enhance the model's performance.
You will record scripted material and assess AI-generated outputs for linguistic accuracy and naturalness. Additionally, you will annotate errors and collaborate with the team to refine prompts and voice design guidelines.
You will record scripted material and assess AI-generated outputs for linguistic accuracy and naturalness. Additionally, you will annotate errors and collaborate with the team to refine prompts and voice design guidelines.
You will record scripted material and assess AI-generated outputs for linguistic accuracy and naturalness. Additionally, you will annotate errors and collaborate with the team to refine prompts and voice design guidelines.
You will record scripted material and assess AI-generated outputs for linguistic accuracy and naturalness. Additionally, you will annotate errors and collaborate with the team to refine prompts and voice design guidelines.
You will record scripted material and assess AI-generated outputs for linguistic accuracy and naturalness. Additionally, you will annotate errors and collaborate with the team to refine prompts and voice design guidelines.
You will record scripted material and assess AI-generated outputs for linguistic accuracy and naturalness. Additionally, you will annotate errors and collaborate with the team to refine prompts and voice design guidelines.
You will record scripted material and assess AI-generated outputs for linguistic accuracy and naturalness. Additionally, you will annotate errors and collaborate with the team to refine prompts and voice design guidelines.
You will record scripted material and assess AI-generated outputs for linguistic accuracy and naturalness. Additionally, you will annotate errors and collaborate with the team to refine prompts and voice design guidelines.
You will record scripted material and assess AI-generated outputs for linguistic accuracy and naturalness. Additionally, you will annotate errors and collaborate with the team to refine prompts and voice design guidelines.
You will record scripted material and assess AI-generated outputs for linguistic accuracy and naturalness. Additionally, you will annotate errors and collaborate with the team to refine prompts and voice design guidelines.
You will record scripted material and assess AI-generated outputs for linguistic accuracy and naturalness. Additionally, you will annotate errors and collaborate with the team to refine prompts and voice design guidelines.
You will record scripted material and assess AI-generated outputs for linguistic accuracy and naturalness. Additionally, you will annotate errors and collaborate with the team to refine prompts and voice design guidelines.
You will record scripted material and assess AI-generated outputs for linguistic accuracy and naturalness. Additionally, you will annotate errors and collaborate with the team to refine prompts and voice design guidelines.
You will record scripted material and assess AI-generated outputs for linguistic accuracy and naturalness. Additionally, you will annotate errors and collaborate with the team to refine prompts and voice design guidelines.
You will record scripted material and assess AI-generated outputs for linguistic accuracy and naturalness. Additionally, you will annotate errors and collaborate with the team to refine prompts and voice design guidelines.
You will converse with the model on software engineering tasks and technical scenarios using Python, verifying logical accuracy and coding fluency. Additionally, you will assess code quality and clarity, capture reproducible error traces, and suggest improvements to our prompt engineering and evaluation metrics.
You will converse with the model on software engineering tasks and technical scenarios using Python, verifying logical accuracy and coding fluency. Additionally, you will assess code quality and clarity, capture reproducible error traces, and suggest improvements to our prompt engineering and evaluation metrics.
You will evaluate AI-generated code, correct errors, and implement optimized Kotlin solutions. Additionally, you will document best practices that demonstrate modern Android design patterns and coding conventions.
You will evaluate AI-generated code, correct errors, and implement optimized Kotlin solutions. Additionally, you will document best practices that demonstrate modern Android design patterns and coding conventions.
You will evaluate AI-generated code, correct errors, and implement optimized Kotlin solutions. Additionally, you will document best practices that demonstrate modern Android design patterns and coding conventions.
You will evaluate AI-generated code, correct errors, and implement optimized Kotlin solutions. Additionally, you will document best practices that demonstrate modern Android design patterns and coding conventions.
You will evaluate AI-generated code, correct errors, and implement optimized Kotlin solutions. Additionally, you will document best practices that demonstrate modern Android design patterns and coding conventions.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to our prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to our prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and document failures to improve AI training. Your expertise will help shape the next generation of AI language models.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to our prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to our prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and document failures. Your expertise will help improve prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to our prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to our prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to our prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to our prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to our prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to our prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to our prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to our prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to our prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and document failures to improve AI training. Your expertise will help shape the next generation of AI for Swedish speakers.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to our prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and document every failure mode. Your expertise will help improve prompt engineering and evaluation metrics for AI training.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to our prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to our prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and document error traces. Your expertise will help improve prompt engineering and evaluation metrics for AI training.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to our prompt engineering and evaluation metrics.
You will converse with the AI model on language scenarios, verify factual accuracy, and document error traces. Your expertise will help improve prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to our prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and capture reproducible error traces. Additionally, you will suggest improvements to our prompt engineering and evaluation metrics.
You will converse with the AI model on language scenarios, verify factual accuracy, and capture reproducible error traces. Additionally, you will suggest improvements to prompt engineering and evaluation metrics.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and document failures to improve AI training. Your expertise will help challenge and refine advanced language models in various aspects of the Chinese language.
You will converse with the model on language scenarios, verify factual accuracy and logical soundness, and document every failure mode. Your expertise will help improve the training data for AI language models.
You will evaluate AI-generated code, correct errors, and implement optimized Kotlin solutions. Additionally, you will document best practices that demonstrate modern Android design patterns and coding conventions.
You will review and assess AI-generated content related to M&A scenarios and evaluate the validity of financial and legal reasoning. Additionally, you will document performance gaps and work closely with the team to refine prompts and evaluation criteria.
You will engage with advanced language models on clinical scenarios and theoretical medical questions, verifying their accuracy and suggesting improvements. Documenting error traces and enhancing model reasoning will be key aspects of your role.
You will engage with advanced language models on clinical scenarios and theoretical medical questions, verifying their accuracy and logical soundness. Additionally, you will document errors and suggest improvements to enhance model reasoning.
You will review AI-generated Kinyarwanda content for linguistic accuracy and fluency, and provide linguistic feedback to improve model comprehension. Your expertise will help refine data and enhance the quality of AI outputs.
You will review and annotate Kyrgyz content, assess AI-generated outputs for accuracy and fluency, and identify and document error patterns. Collaborating with the team, you will refine prompts, evaluation methods, and linguistic guidelines.
You will review and annotate Khmer content, assess AI-generated outputs for accuracy and fluency, and identify and document error patterns. Collaborating with the team, you will refine prompts, evaluation methods, and linguistic guidelines.
You will review and annotate Occitan content, assess AI-generated outputs for accuracy and fluency, and identify and document error patterns. Collaborate with the team to refine prompts, evaluation methods, and linguistic guidelines.
You will review and annotate Tajik content, assess AI-generated outputs for accuracy and fluency, and identify and document error patterns. Collaborating with the team, you will refine prompts, evaluation methods, and linguistic guidelines.
You will review and annotate Welsh content, assess AI-generated outputs for accuracy and fluency, and identify and document error patterns. Collaborating with the team, you will refine prompts, evaluation methods, and linguistic guidelines.
You will review and annotate Uzbek content, assess AI-generated outputs for accuracy and fluency, and identify and document error patterns. Collaborating with the team, you will refine prompts, evaluation methods, and linguistic guidelines.
You will review and annotate Mongolian content, assess AI-generated outputs for accuracy and fluency, and identify and document error patterns. Collaborating with the team, you will refine prompts, evaluation methods, and linguistic guidelines.
You will review and annotate Luo content, assess AI-generated outputs for accuracy and fluency, and identify and document error patterns. Collaborating with the team, you will refine prompts, evaluation methods, and linguistic guidelines.
You will converse with the AI model in Armenian, verifying linguistic accuracy and cultural relevance. Additionally, you will document errors and suggest improvements to enhance the model's performance.
You will review and annotate Asturian content, assess AI-generated outputs for accuracy and fluency, and identify and document error patterns. Collaborating with the team, you will refine prompts, evaluation methods, and linguistic guidelines.
You will review and annotate Māori content, assess AI-generated outputs for accuracy and fluency, and identify and document error patterns. Collaborating with the team, you will refine prompts, evaluation methods, and linguistic guidelines.
You will review and annotate Maltese content, assess AI-generated outputs for accuracy and fluency, and identify and document error patterns. Collaboration with the team to refine prompts, evaluation methods, and linguistic guidelines is also expected.
You will review AI-generated Kinyarwanda content for linguistic accuracy and fluency, and annotate text for tone and meaning. Additionally, you will identify and correct subtle language errors and collaborate with the team to improve prompt design and quality assurance standards.
The Retirement Plan Administration team handles compliance testing, government reporting, and plan terminations for clients. They ensure cross-functional alignment and drive product improvements in retirement plan administration.
Handle year-end testing for DC plans and manage the termination process for retirement plans. Communicate with clients throughout the termination process and draft necessary resolutions.
Provide customer support via phone, email, and live chat, ensuring a positive experience for clients. Assist with account activation, integration troubleshooting, and product-related inquiries.
Provide customer support via phone, email, and live chat, focusing on retirement plan inquiries and technical troubleshooting. Collaborate with internal teams to escalate issues and improve customer service.
Oversee Customer Support Agents and ensure operational excellence in supporting clients with their retirement plans. Provide advanced-level support and handle escalations while maintaining service quality and training the team.
Oversee Customer Support Agents and ensure operational excellence in supporting clients with retirement plans. Provide advanced-level support and handle escalations while monitoring service quality and team performance.
Support coordination of payroll systems setup to ensure implementation is frictionless. Increase client speed to complete onboarding through successful completion of project tasks.
You will support coordination of payroll systems setup to ensure implementation is frictionless. This includes referencing existing documentation and ensuring action items are completed in a timely manner.
As a Specialist on the Plan Documents Team, you will assist with plan design review, drafting plan documents, and plan setup while consulting with various teams on operational considerations. You will also provide support by preparing plan documents, assisting with project management, and performing duties related to plan documents and operational corrections.
As a Specialist on the Plan Documents Team, you will assist with plan design review, drafting plan documents, and plan setup. You will work closely with various teams to ensure compliance and operational considerations are met.
You will evaluate, annotate, and refine AI-generated content in Georgian. This includes reviewing text outputs, correcting errors, and contributing to evaluation guidelines.
You will plan and execute shoots, maintain organized archives, and collaborate with creative teams to ensure consistency in visual tone and message. Your work will help visually define brand identity and contribute to impactful visual storytelling.
You will review AI-generated Kinyarwanda content for linguistic accuracy and fluency, and provide linguistic feedback to improve model comprehension. Additionally, you will annotate text for tone and meaning and collaborate with the team on quality assurance standards.
You will review and assess AI-generated content related to M&A scenarios and evaluate model outputs for accuracy and logic. Collaborating with the team, you will refine prompts and frameworks to improve reasoning and analytical depth.
You will analyze AI-generated legal outputs for accuracy and clarity, and simulate litigation scenarios. Collaborating with the team, you will refine prompts and evaluation standards.
You will analyze AI-generated legal outputs for accuracy, clarity, and logical rigor. Additionally, you will simulate litigation scenarios and collaborate with the team to refine evaluation standards.