Resillion
Senior Automation Tester (Python) – Agentic AI & LLM Systems (Hybrid)
How you stack up
Upload your CV to see how well it fits this job role
?%
Senior Automation Tester (Python) – Agentic AI & LLM Systems (Hybrid)
At Resillion, our culture is built on a mindset of “if you see something, say something” — taking responsibility and speaking up. We expect everyone to adapt and embrace change as the company grows. We recognise the value of every employee and invest in developing their skills so that, as an organisation, we continue to lead our industry.
Above all, it’s a culture driven by passion for what we do and commitment to the bigger picture of the company.
If you want to be part of our journey, this role may be the right fit for you
Job Description
Senior Automation Tester (Python) – Agentic AI & LLM Systems
Location: Glasgow (Hybrid – 3 days per week on‑site)
Start Date: Immediate
Right to Work: Must have full UK working rights (no sponsorship available due to start dates)
Additional Requirement: Must have 3 years of UK address history due to immediate onboarding
Programme: AI / Agentic AI
Seniority: Senior Automation Engineer
Overview
We are seeking a Senior Automation Engineer / SDET with strong Python skills and hands‑on experience testing AI systems, including LLM‑driven applications and agentic AI workflows. You will design and build a scalable automation frameworks, implement modern AI testing strategies, and ensure the quality and reliability of next‑generation conversational and agent‑based AI products.
Reasons to use Rodeo
I’m in my final year doing Economics and I don’t know whether to apply for grad schemes now or do a masters first. What do you think?
Honest answer — it depends on where you want to end up. A lot of top grad schemes (Big 4, civil service, banking) don’t need a masters. Let’s look at the ones you’d be competitive for now, and we can decide if a masters actually adds anything.
Also worth knowing: most autumn 2026 applications are open now. Timing matters more than you think.
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
Graduate Consultant — 2026 Scheme
Why you're a good match
StrongYour economics background and your summer at a regional bank line up with what PwC looks for on the consulting scheme. Applications close in four weeks.
See breakdownIt searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Why you're a good match
You’ve got the grades and the economics background, and your bank internship is exactly the experience this scheme looks for. Apply soon — deadlines close within the month.
Experience fit
Your summer at the bank plus your econometrics coursework map directly to the day-one responsibilities on this scheme — client modelling, market briefings, and deal support.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
This role is ideal for someone who is passionate about automation engineering and has a solid understanding of how AI/LLM systems behave in real‑world scenarios.
Key Responsibilities
Develop and maintain a Python‑based automation framework for validating AI agents (chatbots, voicebots, agentic workflows). Design and execute automated tests for: LLM responses Conversational flows Retrieval‑augmented generation (RAG) End‑to‑end agentic scenarios Apply modern AI testing strategies and evaluation methods (e.g., DeepEval, RAGAS — beneficial but not essential). Integrate automated tests into CI/CD pipelines (Jenkins, GitHub Actions, Azure Repos). Build and manage batch and scheduled executions using Docker, Kubernetes, cron jobs, or similar tooling. Manage code repositories using Git, including branching strategies, version control, and collaborative workflows. Collaborate closely with engineering, AI/ML, and product teams to ensure high‑quality delivery of AI‑powered features.
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Required Skills & Experience
Strong Python development skills, particularly for automation (PyTest preferred). Solid background in test automation and framework design. Experience testing or validating AI/LLM‑based systems (agentic workflows, conversational AI, RAG pipelines, etc.). Understanding of AI evaluation concepts — familiarity with tools like DeepEval or RAGAS is a bonus, not a requirement. Experience implementing CI/CD pipelines and automated batch executions. Strong understanding of source code management, Git workflows, and collaborative engineering practices. Ability to design structured, scalable, and maintainable automation solutions.
Preferred Qualities
Quick learner with a proactive, ownership‑driven mindset. Strong problem‑solving skills and ability to work in fast‑moving AI environments. Comfortable collaborating with cross‑functional teams (AI/ML, engineering, product). Passion for AI technologies and modern testing methodologies.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills