Queen Square Recruitment
AI Evaluation Engineer
London
£600 – £700/day
Posted about 14 hours ago
Get notified of more jobs like this · No spam, ever
How your CV stacks up
1Upload CV
2Analyse CV
3Improve CV
Upload your CV to see how well it fits this job role
?%
AI Evaluation Engineer
Location: London, UK
Work Model: Hybrid (1-2 Days per Week in Office)
Contract Duration: 6 Months
Rate: £600 - £700 Per Day (Inside IR35)
Key Responsibilities
- Define and implement end-to-end evaluation strategies for Generative AI conversational systems, including LLMs, RAG pipelines, and AI agents.
- Establish and maintain evaluation metrics, benchmarks, and quality standards for AI-powered applications.
- Create and manage test datasets, benchmark suites, and evaluation scenarios.
- Benchmark different GPT variants, open-source LLMs, and prompt engineering strategies.
- Develop and maintain automated evaluation pipelines to support continuous testing and quality monitoring.
- Conduct human-in-the-loop evaluations to validate qualitative response quality and business relevance.
- Drive prompt optimization and response quality improvements to improve accuracy and consistency.
- Validate RAG systems and knowledge grounding, ensuring responses are aligned with retrieved information.
- Perform safety, risk, compliance, and Responsible AI testing across AI solutions.
- Analyze model performance and translate findings into actionable recommendations for improvement.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Required Skills
- Strong understanding of Large Language Models (LLMs), RAG architectures, and Prompt Engineering
- Hands-on experience with Python for data analysis, experimentation, and evaluation automation
- Experience designing and implementing AI evaluation frameworks
- Strong knowledge of LLM evaluation metrics and NLP quality assessment
- Experience with benchmarking, scoring, and validating AI model outputs
- Familiarity with evaluation tools such as DeepEval, PromptTools, Ragas, Promptfoo, or similar frameworks
- Experience building automated testing and evaluation workflows
- Strong analytical and problem-solving skills
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Experience Required
- End-to-End Evaluation Framework Design
- Evaluation Metrics Definition & Quality Assessment
- Test Dataset Creation & Benchmarking
- LLM Performance Evaluation and Scoring
- Prompt Testing, Optimization & A/B Evaluation
- RAG Evaluation & Knowledge Grounding Validation
- Hallucination Detection & Response Validation
- Safety, Risk & Compliance Testing
- Python-Based Experimentation Frameworks
- Automation & Evaluation Tooling
Nice to Have
- Experience with Responsible AI principles and governance frameworks
- Experience evaluating Generative AI Agents and Autonomous Workflows
- Knowledge of customer support and retail conversational AI use cases
- Familiarity with LangSmith, MLflow, TruLens, Azure AI Evaluation SDK, or similar platforms
- Experience working within enterprise-scale AI transformation programs
- Understanding of observability and monitoring for AI applications
Trusted by 25,000+ job seekers
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills
Large Language Models
RAG Architectures
Prompt Engineering
Python
AI Evaluation Frameworks
NLP Quality Assessment
Benchmarking
DeepEval
PromptTools
Ragas
Promptfoo
Automated Testing
Hallucination Detection
Responsible AI
Data Analysis
Knowledge Grounding
Location