ConnexAI
AI Reliability & Test Automation Engineer
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
AI Reliability & Test Automation Engineer
About the Role
We are building real-time conversational AI systems powered by LLMs, ASR, and TTS. Our products evolve quickly, with new services, workflows, models, and customer-facing features constantly being developed.
We are looking for an AI Reliability & Test Automation Engineer to work directly within the LLM AI team to build our automated test capability, helping our systems ship safely and confidently.
This is an embedded engineering role focused on the craft of software testing in complex AI systems. You will:
- Design and maintain automated test suites
- Create regression checks
- Develop mocks and CI/CD test gates
- Validate release processes across LLM-powered workflows, APIs, RAG systems, and conversational journeys
You will work closely with Machine Learning Engineers, Data Scientists, Backend Engineers, and external testing teams to ensure new features are testable from the start, enabling thorough evaluation across the wider platform.
Responsibilities
- Build and maintain automated unit, integration, API, and end-to-end test suites for LLM-powered services
- Create regression test coverage for:
- Conversational workflows
- RAG pipelines
- Prompt changes
- Model updates
- Service-level behaviour
- Develop test harnesses, mocks, fixtures, and tools for validating new features
- Integrate automated tests into CI/CD pipelines to provide rapid feedback during development and release
- Work with MLEs, Data Scientists, and backend engineers to:
- Identify failure modes
- Evaluate edge cases
- flag high-risk product behaviours
- Collaborate with external testing teams to connect manual/exploratory validation with automated checks
- Improve testability in Python services by supporting:
- Better interfaces
- Dependency isolation
- Observability
- Support release confidence by creating clear automated checks for:
- Critical user journeys
- System behaviours
- Design load and stress testing frameworks for high-concurrency conversational AI systems
- Leverage production telemetry and failure analysis to evolve automated test coverage
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Requirements
Experience in:
- Software test automation or SDET environments
- Building automated API or backend testing frameworks
- Strong Python, JavaScript, or TypeScript programming
- Integrating automated tests into CI/CD pipelines
- Critical thinking about system behaviour, edge cases, and failure modes
- Strong debugging and problem-solving skills
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Nice to have (but not essential):
- Understanding of modern AI systems, LLMs, or conversational AI products
- Testing distributed systems, APIs, or production platforms
- Experience in fast-paced, high-scale SaaS environments with rapid development
- Testing distributed, asynchronous, or non-deterministic systems
- Collaborating cross-functionally with research and engineering teams
Preferred Backgrounds
- SDETs or QA Automation Engineers with:
- Strong Python and backend testing experience
- Backend Software Engineers passionate about:
- Internal tooling
- CI/CD
- Testing frameworks
- Automation engineers from high-scale SaaS teams proficient in:
- Building test automation for rapid development cycles
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills