SoCode Recruitment
AI Quality Engineer
Get notified of more jobs like this · No spam, ever
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Job Title: AI Quality Engineer (Senior options available too)
Location: Hybrid - 2 days per week in Central London
Salary: Flexible Depending on Experience
Are you an AI Quality Engineer with solid experience of running layered evaluation frameworks, looking to join a business at the forefront of AI-Powered investigative technology?
As the platform continues to scale, the volume of customers, product releases and regulatory challenges is increasing. With AI-generated output forming an important part of the product, maintaining confidence in quality is a complex engineering challenge. You’ll have the opportunity to help shape how AI-driven functionality is evaluated, tested and improved, working closely with engineering and product teams to build confidence in releases.
You’ll work across the evaluation systems and test infrastructure that enable the team to confidently release non-deterministic AI functionality.
This is a hands-on role covering test automation, AI evaluation, CI infrastructure and investigation of complex test and quality issues.
Key Responsibilities:
- Enhance and run layered evaluation frameworks, including automated checks, LLM-as-judge scoring, defined rubrics and targeted human review.
- Maintain versioned golden datasets to ensure evaluations remain reproducible and auditable.
- Monitor key quality signals across investigative outputs, including groundedness, hallucination rate, entity resolution and source quality.
- Run and improve a tiered CI testing model using Python, Playwright and pytest.
- Maintain test coverage across six authenticated personas, three browsers and multi-tenant environments.
- Investigate flaky tests, inconsistent results and changes in evaluation metrics, identifying and addressing root causes.
- Work collaboratively with engineering and product teams to improve quality and confidence in releases.
- The team uses Claude Code across the testing workflow, including selector synchronisation, failure triage and evaluation scoring. You'll need to apply your own judgement when interpreting results rather than relying solely on whether a test has passed or failed.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
There are two potential routes into the role. You may already be a senior engineer with experience across much of the above, or you may have strong test automation foundations and the ability and motivation to develop your experience in AI evaluation.
Essential Experience
- Strong test automation engineering experience.
- Experience with Python or a comparable programming language.
- Experience with a modern automation framework such as Playwright, pytest or similar.
- Strong understanding of CI/CD.
- An analytical and questioning approach to testing and quality.
- The ability to investigate test failures and inconsistent metrics and identify underlying causes.
- Strong communication skills, with the ability to explain complex technical issues to different audiences.
- A collaborative approach to working with engineers and product teams.
- Evidence that you can learn and become competent in technically challenging areas quickly.
- Genuine interest in the challenges involved in testing non-deterministic AI output.
Get help with your application
Your very own career expert that helps elevate your application to the next level.
At a Senior Level:
- It would be beneficial to also have experience of:
- LLM evaluation.
- Rubric design.
- LLM-as-judge approaches.
- AI regression testing.
- Evaluation harnesses.
- Improving quality practices across teams.
Benefits include:
- Employee shares and equity programme, giving you the opportunity to own a meaningful part of the business.
- Private health insurance & Life insurance.
- Unlimited holidays.
- Annual professional development fund to support learning and development.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Location