Xperteez Technology
AI Evaluation Specialist, Remote - Contract
Get notified of more jobs like this · No spam, ever
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Role Title: AI Evaluation Specialist
Role Type: Contract
Location: Remote (US, CA, UK, IE, AU, NZ)
In this role, you'll apply your expertise to help train next-generation AI systems. Your work will shape how models learn, reason, and perform through high-quality, real-world input. No prior experience in AI is required — your domain knowledge is what matters.
Scope of Work
- Evaluate AI-generated outputs against detailed rubrics and defined quality standards, focusing on accuracy, relevance, and adherence to guidelines.
- Apply consistent, impartial judgment across a high volume of examples, ensuring a fair and reliable assessment process.
- Identify reasoning gaps, tool-use failures, or logic errors in AI assistant responses, providing actionable feedback for iterative improvement.
- Produce clear, concise written feedback on both strengths and areas for improvement, directly influencing model refinement and AI adoption practices.
- Participate in discussions regarding rubric interpretation and evolving quality standards, contributing to process optimization and best practices.
- Maintain meticulous documentation of evaluations and recommendations, ensuring transparency and traceability in assessment workflows.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Preferred Qualifications
Get help with your application
Your very own career expert that helps elevate your application to the next level.
- Experience in grading, quality assurance, editorial review, assessment, annotation, or similar fields demanding careful analysis and detailed feedback.
- Advanced, daily use of AI assistants (such as ChatGPT, Claude, or similar) as an essential work and productivity tool.
- Demonstrated ability to synthesize complex information and communicate findings effectively in writing.
- Background in process improvement, rubric development, or operational quality assessment in an enterprise or educational context.
- Strong critical thinking skills with a focus on consistency, integrity, and fairness in evaluations.
- Comfort working independently on large volumes of similar examples while maintaining high attention to detail.
- Collaborative mindset for sharing insights, discussing ambiguous cases, and refining evaluation criteria as models evolve.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Location