Matillion
Staff QA Engineer (AI)
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Are you ready to shape the future of data?
Matillion is the company behind Maia, the AI Data Automation platform.
In the AI economy, demand for data is exploding. Manual data engineering can’t keep up. Maia fixes that by combining 15 years of data expertise with agentic AI to eliminate the manual work that slows data teams down. Organisations like Cisco, DocuSign, and Slack are already using it to deliver real business impact, faster.
We started in Manchester and now we’re global - with teams across the UK, US and India.
We are driven, curious, energetic people who move fast, think big and hold ourselves to a high standard. We’re here to make a dent in the universe bigger than ourselves. And we’re looking for people who want to be part of it.
About the role
We're Matillion, the company behind Maia. Maia is the first AI Data Automation platform that rethinks manual data work by autonomously creating, managing, and evolving data products for humans and AI agents at scale. Quality isn't an afterthought here; it's embedded into everything we ship. As a Staff Quality Engineer, you'll take a senior position within one of our small, focused engineering teams - bringing advanced technical guidance, deep hands-on expertise and a relentless drive to embed quality across the entire development lifecycle.
This is primarily a hands-on technical role, with a secondary focus on technical mentorship and upskilling the team in AI quality practices. You'll set the quality standard for the team, champion automation-first and shift-left testing methodologies, and play a pivotal role in shaping the overall quality journey across Matillion's Engineering department. If you believe quality is everyone's responsibility and you have the experience to prove it - we want to hear from you.
The role sits within the AI team - the group building Maia's core capabilities. Quality here means something sharper than it does elsewhere: our systems are built on large language models, which are non-deterministic by nature, so testing them takes more than traditional pass/fail assertions. You'll help define what "quality" even means for a system like Maia, and build the evals, harnesses and tooling that let us prove it.
Matillion operates on a hybrid basis, with us coming together 2 days a week in our Manchester office to collaborate and drive our mission forward.
What you will be doing
- Building and evolving the eval suites that quantify Maia's performance - including LLM-as-judge test harnesses - so the team can change models, prompts and pipelines with confidence instead of guesswork
- Curating and maintaining the golden datasets our evals run against, and teaching developers across the team how to contribute to and maintain both the evals and the data behind them - quality here is shared, not something that lives solely with QE
- Owning and evolving how we test Maia's agentic workflows - multi-step tool use, reasoning chains and error recovery - a discipline this team has built from the ground up and keeps pushing forward
- Turning eval and test results into visualisations and trend reporting that show us, release over release, whether Maia's capabilities are genuinely moving forward
- Continuously auditing test coverage against real customer usage, so our eval suites stay representative of how Maia is actually used rather than how we assume it's used
- Extending quality practices beyond pre-release testing into production - monitoring live outputs alongside the rest of the team to catch drift and regressions that CI alone won't surface
- Partnering daily with Matillion's Data Science team on the statistical rigour behind our evals - designing sound experiments and correctly reading whether a change is a genuine improvement or just noise
- Stepping back from individual test suites to spot gaps and friction in our broader testing frameworks and methodology - and taking the initiative to revisit, rework or introduce entirely new approaches when the way we test needs to evolve
- Acting as a source of gen AI testing expertise for the wider Engineering organisation - helping quality engineers in other teams stand up evals and learn how to test generative AI features as they build their own AI functionality
- Staying ahead of the curve on generative AI testing - researching new techniques, frameworks and tools on your own initiative, without needing to be asked
- Leading and overseeing the testing of Matillion products - hands-on in the work while mentoring and guiding junior engineers to raise the bar across the team
- Designing and driving robust quality frameworks that span automation, security, accessibility and performance testing - with an automation-first mindset at the core
- Championing shift-left testing across the development lifecycle, embedding quality from design through to deployment via CI/CD pipelines
- Taking a leadership role in cross-functional collaboration - building a culture of quality excellence and elevating standards across the wider Engineering organisation
- Identifying areas for improvement in the quality journey, driving continuous iteration and playing an active role in talent acquisition and team development
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
What we are looking for
- Proven expertise in end-to-end testing of SaaS applications across complex distributed systems - including risk-based, exploratory, regression, security and usability testing approaches
- An understanding of large language models and why their non-deterministic behaviour makes traditional, assertion-based testing insufficient on its own
- Experience building or working with evals and LLM-as-judge techniques to score and quantify the quality of generative AI systems
- A working grasp of statistical thinking - you know that non-deterministic outputs make single-run results noise, and how to design and read evals across samples rather than as simple pass/fail checks
- Deep hands-on experience building and maintaining test automation frameworks (Cypress, Pact or equivalent), with strong proficiency in Java or JavaScript and a solid understanding of CI/CD pipelines
- Experience with at least one major cloud platform (AWS, GCP or Azure) and advanced knowledge of databases and SQL
- A genuine QA mindset - you challenge norms, push back when quality is being compromised and continuously improve how the team works
- Autonomy and inquisitiveness in equal measure - you dig into problems independently, understand causes not just effects, and bring a growth mindset to everything you do
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Compensation
At Matillion, we are committed to providing compensation in line with market standards based on the role, job family, job level and country. This role’s estimated annual salaried pay range for this position is £66,000 - £99,000. The final salary will be based on your relevant skills, experience, and qualifications demonstrated in the hiring process.
More about Matillion
We operate a flexible working culture that promotes work-life balance, with benefits including:
- Company Equity
- 30 days holiday + bank holiday
- 5 days paid volunteering leave
- Private Health Insurance
- Life Insurance
- Pension
- Access to mental health support
Your Safety
Matillion recruiters will only contact you from @matillion.com email addresses. We occasionally work with trusted recruiting partners; they will always tell you they are working on our behalf. We will never ask for money, fees, or bank details at any point in the hiring process. If something feels off, trust that instinct. Don't click any links. Go straight to matillion.com/careers to verify open roles or reach us at talent@matillion.com.
Want to know more?
Don't tick every box? Apply anyway. We hire for potential, not just experience. A member of our Talent Acquisition team will be in touch.
Cannot find anything suitable role right now? We still want to hear from you. Drop us a line at talent@matillion.com.
Find out more about life on #TeamGreen here.
Equal Opportunity Employer
Matillion is an equal opportunity employer. We celebrate diversity and we are committed to creating an inclusive environment for all of our team. Matillion prohibits discrimination and harassment of any type. Matillion does not discriminate on the basis of race, colour, religion, age, sex, national origin, disability status, genetics, sexual orientation, gender identity or expression, or any other characteristic protected by law.
We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses and identifying potential inconsistencies or verification signals in application materials based on available information. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills