TalentCo
Senior AI Evaluation Engineer
Get notified of more jobs like this · No spam, ever
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Senior AI Evaluation Engineer
TalentCo are delighted to be partnering with an established SaaS business at the forefront of secure customer engagement and Conversational AI, looking to hire a high-calibre Senior AI Evaluation Engineer into their growing AI organisation.
The core role of the Senior AI Evaluation Engineer will be to define, build and operate the evaluation systems used to measure the quality, reliability and safety of production RAG, conversational and Agentic AI applications.
You'll work across retrieval, generation, multi-turn conversations, agent reasoning and tool use, combining exploratory investigation with robust Python automation. This isn't a conventional QA or test automation role; you'll define what good AI performance looks like, translate it into measurable criteria, and build the datasets, tooling and release controls needed to evaluate non-deterministic AI systems.
This hire can be based remote-first in the UK, with occasional travel to their HQ (just north of London) as required.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
As Senior AI Evaluation Engineer you will:
- Define evaluation strategies, metrics, acceptance thresholds and release gates across conversational, RAG and Agentic AI applications.
- Build and maintain automated evaluation frameworks in Python using tools such as Ragas and DeepEval.
- Create representative evaluation datasets, golden test cases, adversarial cases and regression suites across complex AI workflows.
- Evaluate RAG and agent behaviour across retrieval quality, grounding, hallucination, tool use, reasoning, memory, safety and multi-turn conversations.
- Integrate evaluation into CI/CD workflows and work closely with AI, Software Engineering, Product and Security teams to identify failures and improve system performance.
We want to hear from you if:
- You have a strong Software Engineering, AI Evaluation, Quality Engineering or Test Automation background, with significant experience evaluating production RAG and Agentic AI systems.
- You're highly proficient in Python and comfortable building maintainable evaluation libraries, test harnesses and automated evaluation pipelines.
- You have hands-on experience with Ragas, DeepEval or comparable LLM evaluation frameworks.
- You understand modern LLM applications including embeddings, vector retrieval, reranking, prompting, tool calling and agent orchestration.
- You have experience with AI security, red-teaming or responsible AI, ideally within regulated environments.
Get help with your application
Your very own career expert that helps elevate your application to the next level.
What do they offer?
- Base salary of £75k–£95k, dependent on experience
- Comprehensive benefits package including bonus and Share Incentive Scheme
- Remote-first working with flexible arrangements
- 25 days holiday, private healthcare options and pension scheme
- Opportunity to take ownership of AI evaluation across a growing Conversational AI platform
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills
Location