Lorien
Test Lead - GenAI & LLM Testing
Get notified of more jobs like this · No spam, ever
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Senior QA Automation Engineer / SDET - GenAI & LLM Testing
6 Month Contract
Banking Client
Remote / Hybrid (UK)
Competitive Day Rate
We're supporting a leading banking client on a large-scale AI transformation programme and are looking for an experienced Senior QA Automation Engineer / SDET with a strong background in GenAI and LLM testing.
This role will focus on defining and implementing automated testing frameworks for AI-driven products, ensuring the quality, reliability and safety of LLM-powered solutions deployed within a highly regulated banking environment.
Key Responsibilities
- Design and develop scalable automated test frameworks using Python.
- Create test strategies from architecture and solution design documentation.
- Test RESTful APIs and microservice-based applications.
- Define and execute testing approaches for event-driven architectures and Kafka-based systems.
- Validate GenAI and LLM solutions through functional, non-functional and behavioural testing.
- Develop automated prompt testing and evaluation frameworks.
- Verify AI guardrails, safety controls and compliance requirements.
- Implement validation approaches for non-deterministic LLM outputs.
- Build and maintain contract tests using Pact Flow.
- Collaborate closely with Engineering, Architecture, Data Science and Product teams.
- Support CI/CD pipelines and quality engineering best practices.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Required Skills & Experience
Core Automation
- Strong Python 3.x development skills with experience building testing frameworks from the ground up.
- Extensive experience with Pytest.
- Strong API testing experience.
- Experience creating test strategies and quality approaches from architectural diagrams and solution designs.
Cloud & Platform
- Working knowledge of AWS services including:
- Lambda
- Bedrock
- S3
- EKS
Event-Driven Systems
- Experience testing Kafka-based architectures.
- Strong understanding of asynchronous messaging and event-driven systems.
Get help with your application
Your very own career expert that helps elevate your application to the next level.
GenAI / LLM Testing (Essential)
- Proven experience testing LLM and GenAI applications.
- Prompt testing and prompt evaluation.
- Non-deterministic output validation.
- AI guardrail verification and safety testing.
- Hallucination and response quality assessment.
- Experience establishing quality standards for AI solutions.
LLM Evaluation Tooling
- Experience with one or more of:
- DeepEval
- Promptfoo
- RAGAS
- LangSmith
- Similar LLM evaluation frameworks
Contract Testing
- Hands-on experience with Pact Flow.
- Consumer-driven contract testing within distributed systems.
Desirable
- Banking or Financial Services experience.
- Experience working in highly regulated environments.
- Exposure to RAG architectures.
- Knowledge of LLM observability and monitoring.
- Kubernetes experience.
- CI/CD pipeline integration.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills
Location