GlobalJobFeed
Machine Learning Engineer
Get notified of more jobs like this · No spam, ever
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Remote (Northern Ireland) | Full Time | £90,000 to £140,000 base plus equity
About the Role
This is a new role and the first dedicated evaluation hire, so you will define how the business measures the quality of its AI. The product is an AI agent, and every change to a model, prompt or piece of agent logic can quietly make it better or worse. Your job is to know which, before customers do.
You will work closely with the engineers building agent capabilities and with the team's senior data scientist, and report to the Head of Data Science. The team is small, so breadth matters as much as depth, and you will regularly pick up work outside your core specialism.
The people who thrive here are hands-on and curious about the business, not just the technology. You will be comfortable in the data, confident learning on the job and calm when priorities shift. You will know when an evaluation is good enough to ship and when it needs more work. Cost is a real constraint in a start-up, so you will think about what each LLM call and pipeline costs as naturally as how accurate it is.
Key Responsibilities
- Design evaluation frameworks and metrics covering accuracy, safety, latency and cost across agent and LLM systems
- Build benchmark eval sets that reflect real customer scenarios and edge cases
- Develop automated scoring pipelines using rubric-based grading and LLM-as-judge techniques
- Calibrate automated judges against human review so the team can trust the numbers
- Stand up regression suites that catch quality drops from model, prompt or agent logic changes before release
- Create dashboards that track model and agent quality over time and across releases
- Investigate failure modes in multi-step agent behaviour and prioritise fixes with the engineering team
- Partner with the senior data scientist on deeper statistical analysis of results
- Balance evaluation coverage against compute and API cost
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
What You'll Need
- 5+ years in ML engineering, NLP or applied data science, with hands-on experience of LLM or agent-based systems
- Practical experience building or operating evaluation frameworks, automated scoring or benchmark systems for ML or LLM outputs
- Strong Python and experience building data pipelines for evaluation datasets
- Solid understanding of NLP and modern LLM capabilities, including prompting techniques, agentic workflows and retrieval
- A strong statistics foundation, including sampling, variance and judging whether a change in results is meaningful
- Ability to design metrics that represent real-world quality, not just benchmark scores
- Hands-on experience with GCP (Vertex AI, BigQuery), or equivalent experience with another major cloud ML platform
- A degree in Computer Science, Machine Learning, Statistics or a related field, or equivalent practical experience
- Right to work in the UK
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Nice to Have
- Experience with LLM-as-judge techniques, rubric design or human-in-the-loop evaluation programmes
- Familiarity with agent architectures and the failure modes specific to multi-step agentic systems
- Experience operating evaluation systems at scale in production
Why Apply?
- £90,000 to £140,000 base salary plus equity in a business heading into its next funding round
- Fully remote, with a requirement to be based in Northern Ireland
- Async-first culture that trusts you to manage your own time and deliver
- Build a function from scratch, with direct influence over how the product's AI is measured and released
- Real breadth: work across engineering, data and ML rather than in a narrow lane
- A clear, transparent five-stage interview process, with AI tools actively encouraged in the take-home exercise
- A long-term career home in a company investing seriously in its data and ML function
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Location