AI Evaluation Specialist

Role Title: AI Evaluation Specialist

Role Type: Contract

Location: Remote (US, CA, UK, IE, AU, NZ)

In this role, you'll apply your expertise to help train next-generation AI systems. Your work will shape how models learn, reason, and perform through high-quality, real-world input. No prior experience in AI is required — your domain knowledge is what matters.

Scope of Work

• Evaluate AI-generated outputs against detailed rubrics and defined quality standards, focusing on accuracy, relevance, and adherence to guidelines.
• Apply consistent, impartial judgment across a high volume of examples, ensuring a fair and reliable assessment process.
• Identify reasoning gaps, tool-use failures, or logic errors in AI assistant responses, providing actionable feedback for iterative improvement.
• Produce clear, concise written feedback on both strengths and areas for improvement, directly influencing model refinement and AI adoption practices.
• Participate in discussions regarding rubric interpretation and evolving quality standards, contributing to process optimization and best practices.
• Maintain meticulous documentation of evaluations and recommendations, ensuring transparency and traceability in assessment workflows.

Preferred Qualifications

• Experience in grading, quality assurance, editorial review, assessment, annotation, or similar fields demanding careful analysis and detailed feedback.
• Advanced, daily use of AI assistants (such as ChatGPT, Claude, or similar) as an essential work and productivity tool.
• Demonstrated ability to synthesize complex information and communicate findings effectively in writing.
• Background in process improvement, rubric development, or operational quality assessment in an enterprise or educational context.
• Strong critical thinking skills with a focus on consistency, integrity, and fairness in evaluations.
• Comfort working independently on large volumes of similar examples while maintaining high attention to detail.
• Collaborative mindset for sharing insights, discussing ambiguous cases, and refining evaluation criteria as models evolve.