Microsoft AI
Senior Network Production Engineer, AI Supercomputing
Get notified of more jobs like this · No spam, ever
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Overview
We are building and operating frontier-scale AI supercomputers used to train the world’s most advanced models. The backend network is a critical part of the machine: a single degraded NIC, switch, link, or path can reduce training performance or disrupt jobs spanning thousands of GPUs.
We are looking for a senior, hands-on Network Production Engineer to own the production health and operations of our multi-rail Ethernet backend network built on MRC and RDMA technologies. You will bridge network engineering, distributed systems, hardware health, and production operations to ensure the network delivers predictable performance at extreme scale.
Your mission is simple: make the network invisible to researchers by maximizing large-job success, minimizing performance degradation, and recovering quickly when failures occur.
Responsibilities
Own Production Network Health
- Own the availability, performance, and operational readiness of the MRC Ethernet backend network.
- Define and operate service-level indicators for packet loss, congestion, link health, path diversity, bandwidth, tail latency, collective performance, and job impact.
- Detect slow or degraded components before they cause training failures or reduce model FLOPs utilization.
- Build health models that correlate switch, NIC, host, topology, and application telemetry.
Operate the Network at Frontier Scale
- Participate in on-call rotation and lead the response to high-severity network incidents.
- Diagnose failures spanning GPUs, NICs, switches, cables, firmware, drivers, network operating systems, MRC, NCCL, Kubernetes, and training workloads.
- Develop automated mitigation mechanisms, including path avoidance, node quarantine, workload relocation, and safe component remediation.
- Create operational procedures for maintenance, upgrades, rollback, capacity expansion, and topology changes.
Improve Large-Job Reliability and Performance
- Work directly with training teams to investigate collective-performance degradation, stalls, timeouts, job restarts, and unexplained MFU loss.
- Translate low-level network signals into clear job-level impact.
- Establish network qualification gates for admitting nodes and racks into large training pools.
- Run failure-injection and resilience testing to validate behavior under link, NIC, switch, and path failures.
- Improve placement and routing policies for jobs spanning racks, rows, and network planes.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Automate Fleet Operations
- Build production-quality software and automation for network validation, monitoring, diagnosis, remediation, and fleet-wide change management.
- Replace manual investigation with deterministic workflows and actionable alerts.
- Automate firmware, driver, configuration, and network operating-system compliance checks.
- Maintain authoritative network inventory, topology, and configuration state.
- Reduce mean time to detect, isolate, mitigate, and permanently resolve failures.
Drive Cross-Company Execution
- Lead technical investigations involving internal infrastructure teams, cloud and datacenter operators, Azure Networking, NVIDIA, switch and NIC teams, and network-software partners.
- Own incidents through resolution, even when the underlying failure crosses organizational boundaries.
- Produce clear root-cause analyses with corrective actions, owners, and completion dates.
- Convert recurring production failures into requirements for future network, server, and accelerator architectures.
- Create runbooks and train engineers across regions to provide reliable round-the-clock coverage.
Qualifications
Required Qualifications:
- Bachelor’s Degree in Computer Science, or related technical discipline AND 4+ years technical experience in Site Reliability Engineering, DevOps, or Infrastructure Engineering OR equivalent experience
- Experience operating large-scale datacenter or high-performance computing networks in production environments.
- Experience with Ethernet, RDMA, RoCEv2, congestion control, routing, load balancing, and lossless or near-lossless network design.
- Experience diagnosing failures across switches, NICs, hosts, drivers, firmware, and distributed applications.
- Experience working with Linux systems and programming in Python, Go, C++, or another systems-oriented programming language.
- Experience building monitoring, automation, diagnostic, or remediation systems for production infrastructure.
- Experience leading high-severity incident response and coordinating resolution across multiple engineering teams.
- Experience analyzing telemetry to identify and validate root causes of infrastructure or network issues.
- Ability to participate in a global on-call rotation.
Preferred Qualifications:
- Master’s Degree in Computer Science, or related technical discipline AND 2+ years technical experience in Site Reliability Engineering, DevOps, or Infrastructure Engineering OR equivalent experience
- Experience operating GPU training clusters or other tightly coupled distributed-computing systems.
- Knowledge of MRC or other multi-rail, multi-plane, multipath Ethernet transports.
- Experience with NCCL, collective communications, CUDA, GPUDirect RDMA, and distributed training frameworks.
- Familiarity with ConnectX-class NICs, modern Ethernet switch ASICs, SONiC, SAI, and switch telemetry.
- Experience with InfiniBand and the operational differences between InfiniBand and Ethernet-based AI fabrics.
- Familiarity with Kubernetes, Slurm, workload placement, and cluster-health certification.
- Experience with streaming telemetry, gNMI, Prometheus, Datadog, Kusto, eBPF, or equivalent observability systems.
- Experience managing fleet-wide firmware, driver, or network operating-system changes.
- A track record of improving reliability through automation rather than recurring manual intervention.
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Compensation
Software Engineering IC5 - The typical base pay range for this role across United Kingdom is £ 93,500.00 - £ 161,800.00 per year. Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here:
This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.
Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Location