Get notified of more jobs like this · No spam, ever
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
About the Role
We’re working with a fast-growing AI company looking for a QA Engineer to own how the quality of its AI products is measured, tested, and improved. This isn’t traditional QA — you’ll work at the intersection of AI evaluation, software engineering, and product quality, building the frameworks, datasets, and automated tests that determine whether LLM-powered features are genuinely accurate, reliable, and ready for production. You’ll work closely with engineering and product teams to identify where AI systems fail, understand why, and turn those findings into measurable improvements.
Must Haves
- Strong software engineering or technical background, ideally with hands-on experience working with AI/ML systems
- Experience building testing, evaluation, or quality frameworks for AI-powered products
- Strong Python or similar programming experience, with the ability to build evaluation tooling and automation
- Good understanding of LLMs and Generative AI, including how model outputs should be tested and evaluated
- Experience creating evaluation datasets, test cases, and regression suites for complex AI features
- Ability to analyze model outputs, identify failure patterns and root causes, and turn findings into measurable improvements
- Strong understanding of accuracy, reliability, and consistency when evaluating AI systems
- Excellent analytical and problem-solving skills, particularly when dealing with ambiguous or difficult-to-measure problems
- Comfortable working closely with Engineering and Product teams and communicating technical findings clearly
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Nice to Haves
Get help with your application
Your very own career expert that helps elevate your application to the next level.
- Experience evaluating RAG / retrieval systems, including the quality and relevance of retrieved information
- Experience testing AI agents, tool-calling, or multi-step AI workflows
- Knowledge of different LLM evaluation methodologies, metrics, and benchmarking approaches
- Experience implementing automated AI regression testing within development or CI/CD workflows
- Experience with human-in-the-loop evaluation alongside automated testing
- Knowledge of prompt evaluation and comparing changes across different models or prompts
- Experience building internal AI evaluation tooling, dashboards, or monitoring
- Previous experience working with AI products where accuracy and reliability are particularly important
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Location