Across the wider product, you'll:

• Extend evaluation across important customer journeys, APIs, backend services and UI.
• Use OpenTelemetry, including its Semantic Conventions, to make benchmark environments representative and portable.
• Use product telemetry and correlate benchmark results with metrics, logs and traces to understand why failures occur.
• Build end-to-end measurements focused on customer outcomes rather than isolated components.
• When we change a model, prompt, tool or agent workflow, we want to know what became better, what became worse and why — including the impact on quality, reliability, latency and cost.

Turn what we learn into continuous improvement

• Turn failures and real-world problems into new evaluation scenarios.
• Identify recurring failure patterns and capability gaps.
• Test potential improvements against established baselines, holdouts and unseen scenarios.
• Detect regressions, benchmark overfitting and improvements that don't generalise.
• Longer term, this evaluation system becomes the harness for controlled self-improvement: identifying weaknesses, testing potential changes and objectively determining whether they should be retained. That's the direction of travel rather than the first year's work, but it's why we're building this properly.

Work with the rest of engineering

• Work directly with AI, software, platform and SRE engineers to investigate findings and improve the product.
• Make evaluation failures clear, reproducible and actionable.
• Make straightforward fixes yourself where that's the most efficient approach.
• Build tooling that makes evaluations easy for other engineers to create, run and understand.
• Use AI-assisted engineering where it improves the speed or quality of your work.
• Benchmarking should provide continuous feedback that helps engineering improve the product. This role is not a release gatekeeper.

What We're Looking For

Above everything else: the ability to take an ambiguous technical problem, develop an approach and deliver a working system independently.

We're more interested in demonstrated ability than an exact number of years, but we'd generally expect around 3+ years of relevant technical experience. Your background might be as an AI engineer, software engineer, SRE, platform engineer, performance engineer, SDET or similar.

Alongside that, we're looking for:

• Practical experience working with LLMs, AI agents or AI evaluation.
• Strong software engineering skills, particularly Python or a similar language.
• Experience building automated evaluation, benchmarking, testing or experimentation infrastructure.
• Experience creating synthetic workloads, datasets or evaluation scenarios.
• An understanding of non-deterministic evaluation, including repeated measurement, variance and distinguishing meaningful changes from noise.
• The ability to turn complex system behaviour into measurable criteria.
• Comfort working across APIs, distributed systems and multiple layers of a software product.

Desirable

Any of the following would be useful, but none are required:

• Agentic AI evaluation, tool use and multi-step workflows.
• LLM evaluation frameworks and model-based evaluation techniques.
• Automated experimentation or self-improving systems.
• Dataset, ground-truth and holdout evaluation design.
• Statistical experimentation and performance benchmarking.
• OpenTelemetry, metrics, logs and distributed tracing.
• SRE, incident response or observability.
• Production SaaS and distributed systems.

What Success Looks Like

We have the beginnings of an evaluation harness, but the design and expertise are what we're hiring for.

We'd expect the first few months to go into the core evaluation harness and a starting corpus for Coworker's investigation quality, then extend outward across the rest of the product as that proves itself. How you sequence it is your call.

By six months, we should be able to objectively answer questions such as:

• Is Coworker getting better at investigating operational problems?
• Where does it perform well or poorly, and why?
• Are its conclusions supported by the right evidence?
• How do different models, tools and agent configurations compare?
• What quality, latency and cost trade-offs are we making?
• How do we compare against relevant external benchmarks?
• Have improvements introduced regressions elsewhere?
• Where should we focus improvement next?

Our evaluation corpus should keep growing as we encounter new problems.

Success isn't measured by the number of tests written or percentage test coverage. It's measured by our ability to understand how well OpsPilot is doing its job, where it isn't, and whether the changes we're making are actually making it better.

What We Offer

A small company rather than a large one, with the trade-offs that implies. Under ten people in engineering, a flat structure, and decisions made in a conversation rather than across three meetings. You'll have genuine influence over how this is done, and very little bureaucracy to work through to get there.

• Fully remote within the UK.
• Flexible working hours.
• 25 days holiday plus bank holidays.
• Real autonomy over your technical approach and how you deliver the role.

Our Interview Process

Straightforward: usually two or three conversations, with no technical coding tests.

If this sounds like the kind of challenge you're looking for, we'd love to hear from you.