How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Founding team · Full-time · Hybrid (London) · Meaningful equity
About us
Science runs on trust, but the machinery meant to produce that trust is failing. Peer review is slow, opaque, inconsistent, and increasingly gamed. And a flood of AI-generated research is about to overwhelm a system that was already straining. No one is the credible, independent arbiter of research quality. We think there should be one.
OxSci is building a credit rating agency for science: a certification layer that combines AI with expert peer review, so researchers, institutions, and AI developers can assess research quality quickly and at scale. We're early, focused, and well-resourced.
The open scientific question at the heart of this company is yours to own: where does the frontier of AI peer review actually stand? What can AI reviewers catch that human experts miss, what do they still get wrong, and how do you prove it rigorously? The standards we set now, including what "quality" even means and how we demonstrate our AI reviewers are actually good, will define both the company and, we believe, the field.
Why this role is unusual for a researcher
If you work on LLM evaluation or AI for science, you know the pattern: the most important studies (expert-annotated meta-evaluation of AI reviewers on real papers) are one-off efforts, because recruiting dozens of expert scientists as annotators takes months and can't be repeated. At OxSci, that study design is our daily operating loop.
- Expert annotation as infrastructure, not a bottleneck. We operate a standing, paid network of expert reviewers producing structured reviews of real submissions, alongside AI reviews of the same manuscripts. The human-AI comparison most labs can only run once, you run continuously, at growing scale.
- Define the benchmark, not just run one. There is no accepted standard for meta-evaluating AI reviewers. The evaluation you design here is positioned to become the reference point for the field.
- Publish and speak. We want your findings in the open literature and on conference stages. You'll represent OxSci publicly in the AI-review and LLM-evaluation research community; our scientific credibility is built on your work being visible and defensible.
- Your research artifact runs in production. Your benchmarks and evaluator improvements ship into a live review system serving real researchers, then feed back new data. At a big lab you'd be one name on a twenty-author paper; here you own the agenda, the data, and the standard.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
About the founders
OxSci was founded by Shumiao Ouyang, Associate Professor of Finance at Saïd Business School, University of Oxford, and Fellow in Management at Wadham College. His research spans fintech, digital payments, and AI; he holds a PhD in Economics from Princeton and degrees from Peking and Tsinghua Universities. He started OxSci out of a conviction that the systems for judging scientific quality are overdue for reinvention, and you'd be building it with him directly.
The technical side is led by OxSci's cofounder, a senior tech lead from one of the world's largest technology companies, with deep experience building and operating systems at global scale. You'd work with both founders day to day, shaping the evaluation methodology and research culture from the ground up.
What you'll do
- Own the research agenda on AI-reviewer evaluation. Track the frontier (AI-scientist, automated-review, LLM-as-a-judge, and scholarly-NLP literature), position our system against it, and decide what we measure next and why.
- Design meta-evaluations that expose weaknesses, not just measure agreement. Build fine-grained, criticism-level evaluations of AI review agents (correctness, factual grounding, significance, sufficiency of evidence, hallucination rate, and venue/journal matching) that reveal where and why they fail, going beyond verdict-matching.
- Run expert-annotation studies at scale. Design the protocols, rubrics, inter-annotator agreement, and statistics needed to compare AI and human reviewers credibly, including head-to-head evaluations against other AI review systems, and defend the numbers to a skeptical scientific audience.
- Build a living taxonomy of AI-reviewer failure modes such as subfield blind spots, long-context degradation, over-anchoring, and spurious criticism, and turn each into a regression benchmark that guards against backsliding as models and prompts change.
- Calibrate the combined rating. Define quality-scoring rubrics for human review reports and calibrate how expert and AI judgment fuse into a single, defensible rating: the core of what universities and publishers buy from us.
- Close the loop. Translate benchmark findings into concrete improvements to our review agents (retrieval, context engineering, orchestration, model choice) and prove the gains with the same rigor you used to find the gaps.
Get help with your application
Your very own career expert that helps elevate your application to the next level.
What we're looking for
- A PhD (or near completion) in CS, ML, NLP, or a related field, or an equivalent research track record. You're likely already working on LLM evaluation, LLM-as-a-judge, AI for science, automated peer review, or a nearby frontier.
- A fascination with the boundary between AI and human reviewers. What each catches that the other misses, and conviction that mapping it rigorously is how trustworthy peer review gets built.
- A track record of rigorous evaluation of ML/LLM systems: benchmark or eval-framework design, evaluator/judge models, expert-annotation study design, hallucination and factuality measurement, uncertainty quantification, or RAG evaluation. Bonus if you've built evaluations that score individual criticisms rather than just verdicts.
- Fluency in evaluation methodology and statistics: sampling, inter-annotator agreement, significance testing, and the discipline to distinguish a real effect from a lucky prompt.
- Strong Python and hands-on habits. You build the eval harnesses and pipelines yourself, not just spec them, with enough LLM-application fluency (RAG, tool calling, orchestration) to turn a finding into a shipped improvement.
- Bonus: publications in NLP/ML evaluation or automated peer review; open-source benchmarks or evaluator models the community actually uses; experience with scholarly content at scale.
What we offer
- Founding seat with meaningful equity and a direct line to the founders
- Ownership of a genuinely open research question, with encouragement to publish and present the work
- A standing expert-reviewer network as your annotation infrastructure
- A proprietary, growing dataset of paired human and AI reviews of real submissions
- The rare chance to define the standard by which AI reviewers themselves are judged
- Genuinely competitive pay; equity discussed openly
- Generous LLM token budget for your daily work
- Flexible working hours; fast personal growth with broad ownership from day one
How to apply
- Location: Hybrid (London)
- Visa: we can sponsor work visas for the right candidate.
- To apply: send your CV to recruit.uk@oxsci.ai, and include something you've built or written: a benchmark, an eval, a paper, or any project you'd love to talk about.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills
Location