Microsoft AI
Principal AI Evaluations Platform Engineer
Get notified of more jobs like this · No spam, ever
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Job Title
As Microsoft continues to push the boundaries of AI, we are looking for passionate individuals to work with us on some of the most interesting and challenging AI questions of our time. Our vision is bold and broad: to build systems with true artificial intelligence across agents, applications, services, and infrastructure. It is also inclusive: we aim to make AI accessible to consumers, businesses, and developers so that everyone can realize its benefits.
Microsoft AI (MS AI) is seeking an experienced engineer to help build and operate the evaluation platform that supports large-scale model training and development. We’re looking for someone who combines strong systems engineering fundamentals with operational excellence and who can build reliable evaluation infrastructure at scale. This role will work closely with researchers and training teams across our European offices to ensure evaluations are reliable, efficient, and available when teams need them.
We seek a versatile engineer who can build solutions that stand the test of time and who brings positive energy, empathy, and kindness to the team while remaining highly effective in a fast-paced environment.
Responsibilities
- Serve as a primary engineer for the evaluation platform during European hours, providing dedicated coverage for the evaluation stack supporting large-scale model training.
- Develop and extend core evaluation platform capabilities, including benchmark configurations, problem sets, graders, and evaluation runners.
- Improve evaluation throughput and scheduling efficiency.
- Build and maintain monitoring, alerting, and dashboards to proactively detect evaluation degradation.
- Partner closely with researchers and training teams across European offices to define evaluation requirements, coordinate launches, and unblock experiments.
- Contribute to technical design and code reviews across the evaluation platform stack, and maintain high-quality operational documentation and on-call runbooks.
- Embody our culture and values.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Required Qualifications
- Bachelor's Degree in Computer Science, Engineering, Mathematics, or a related field AND 4+ years of technical engineering experience with coding in languages including, but not limited to, Python, C++, Go, or Rust;
- OR Master's Degree in Computer Science, Engineering, Mathematics, or a related field AND 2+ years of technical engineering experience with coding in languages including, but not limited to, Python, C++, Go, or Rust;
- OR equivalent experience.
- Demonstrated experience operating production distributed systems, including participation in a formal on-call rotation.
- Strong proficiency in Python, including experience working in a large, shared, typed codebase.
- Experience debugging distributed system failures across the stack, including scheduling, resource exhaustion, networking, and process-level faults.
- Willingness and availability to participate in a scheduled on-call rotation covering European business hours.
Preferred Qualifications
- Experience with GPU-based workloads, distributed training, or large-scale inference infrastructure.
- Experience with distributed compute frameworks and schedulers, such as Ray, Kubernetes, Slurm, or comparable systems.
- Experience with observability and alerting tooling, such as Datadog, Prometheus, or Grafana.
- Familiarity with LLM evaluation, benchmarking, or reinforcement learning training infrastructure.
- Track record of reducing operational burden through systematic reliability improvements, automation, or tooling.
- Ability to work effectively in a fast-paced environment, manage multiple priorities, and adapt to changing requirements and deadlines.
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Compensation
The typical base pay range for this role across Switzerland is CHF 183,800.00 - CHF 309,700.00 per year. Certain roles may be eligible for benefits and other compensation.
Find additional benefits and pay information here: https://careers.microsoft.com/v2/global/en/corporate-pay/switzerland-corporate-pay
Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: https://careers.microsoft.com/us/en/us-corporate-pay
Application
This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.
Equal Opportunity Employer
Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances.
If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Location