Braintrust
Senior Application Security Engineer - AI Code Evaluation
Get notified of more jobs like this · No spam, ever
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Help a top AI lab evaluate and improve large language models through security-focused coding tasks
Bring your software-engineering judgment and hands-on security experience to work involving vulnerabilities, exploit verification, and security patches.
This is a contracting engagement, with potential for a longer-term engagement. Remote candidates in the selected countries are eligible.
What you will do
- Evaluate coding tasks involving software vulnerabilities, exploit verification, and security patches.
- Create high-quality coding prompts and reference answers for benchmark-style problems.
- Evaluate model outputs for code generation, refactoring, debugging, and implementation.
- Identify and document model failures, edge cases, and reasoning gaps.
- Compare private language models with leading external models.
- Build or configure coding environments for evaluation and reinforcement learning.
- Follow detailed annotation and evaluation guidelines consistently.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
What you bring
- At least five years of professional software-development experience and strong Python skills.
- Hands-on experience with vulnerability research, exploit reproduction or verification, or implementing, backporting, or validating security patches.
- The ability to apply structured evaluation criteria and write clear technical feedback.
- Fluency in written and spoken English.
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Helpful, not required
- Professional code review, coding annotation, LLM/code evaluation, or benchmark design.
- Knowledge of another programming language.
- Team leadership or mentoring experience.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Location