How your CV stacks up
Upload your CV to see how well it fits this job role
?%
AI Test Engineer
About Concept Quality Reply: Concept Quality Reply is a QA and software testing company focused on delivering high-quality digital solutions. We provide governance and production monitoring to ensure ongoing performance and compliance after release. Through advanced test automation and AI-driven testing strategies, we help organizations accelerate development, reduce risk and ensure reliability across the entire software lifecycle. Our goal is to transform QA into a strategic driver of innovation and efficiency.
Role Overview: We are looking for a mid/senior-level AI Testing & Engineering Specialist with strong Python expertise and solid experience in AI, Data Engineering, or Test Automation. The role is focused on building and evolving a next-generation testing framework for generative AI agents, including evaluation pipelines, synthetic data generation, observability layers, and quality metrics for LLM-based systems. You will work across AI engineering, data pipelines, and quality automation, contributing both to architecture design and hands-on implementation of scalable testing and evaluation systems. This role is suited for someone who can operate independently, contribute to technical decisions, and help shape standards for AI system validation in production environments.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Responsibilities: Design and develop Python-based frameworks for testing and evaluating AI agents and LLM-based systems Contribute to architecture design for evaluation pipelines, observability, and data-driven testing systems Build and maintain tools for test data generation (including synthetic and adversarial datasets) Define and implement evaluation strategies and quality KPIs for AI behavior (accuracy, robustness, bias, consistency, hallucination rate, etc.) Integrate LLM evaluation tools, CI/CD pipelines, and external AI platforms Support continuous testing, benchmarking, and production monitoring of AI systems Collaborate with AI engineers, data engineers, and QA teams to improve system reliability and scalability
About the candidate: 2–5+ years of experience in Software Engineering, Test Automation, AI Engineering, or Data Engineering Strong proficiency in Python and data/AI libraries (e.g., pandas, numpy, PyTorch or similar) Solid understanding of LLMs, NLP concepts, or generative AI systems Experience with test automation frameworks (e.g., PyTest, Playwright) and CI/CD pipelines Familiarity with data pipelines, dataset design, or feature engineering is a strong plus Experience with evaluation metrics for NLP/AI systems (BLEU, ROUGE, embedding-based metrics, or custom scoring approaches) Ability to design scalable systems and work autonomously on complex technical problems Strong analytical mindset and interest in AI system quality, reliability, and governance
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Reply is an Equal Opportunities Employer and committed to embracing diversity in the workplace. We provide equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type regardless of age, sexual orientation, gender, identity, pregnancy, religion, nationality, ethnic origin, disability, medical history, skin colour, marital status or parental status or any other characteristic protected by the Law.
Reply is committed to making sure that our selection methods are fair to everyone. To help you during the recruitment process, please let us know of any Reasonable Adjustments you may need.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills
Location