NVIDIA Mission Control 2.5 Brings Autonomous Recovery to Kubernetes

From Slurm to Kubernetes

NVIDIA says large-scale AI training has changed the operating model for data centers. Modern AI factories run distributed training jobs across hundreds of nodes for days or weeks. At that scale, hardware anomalies, software errors, network timeouts and resource contention must be managed proactively. The company argues that manual intervention, static checkpointing and reactive monitoring do not scale when organizations run dozens of concurrent multi-node workloads on shared infrastructure.

The autonomous recovery engine in Mission Control is NVIDIA's answer to that problem. With version 2.5, the service now covers Kubernetes workloads managed through NVIDIA Run:ai, in addition to Slurm. NVIDIA describes the goal as moving AI factory operations from reactive maintenance to proactive, self-healing infrastructure.

A three-layer control loop

NVIDIA frames the system as a control loop with three distinct roles rather than a single recovery feature.

• Kubernetes scheduler: In Mission Control, NVIDIA Run:ai is the default Kubernetes scheduler and is included in the software. It manages Kubernetes-based AI workloads, queues and policies, and places jobs on available capacity.
• Autonomous job recovery service: This is the workload-context layer. It watches running jobs, tracks where each one executes, and uses failure history and job topology to classify which recovery path makes sense. The resource manager integration then turns that guidance into a restart, retry, re-queue or placement action.
• Autonomous hardware recovery service: This is the infrastructure-health layer. It monitors nodes and GPUs, removes anomalies from scheduling when needed, and runs diagnostics or remediation so hardware problems do not spread into new jobs.

According to NVIDIA, keeping these responsibilities separate makes the model portable. The job recovery service does not need to repair hardware, and the hardware recovery service does not need to understand every training framework. The scheduler can also steer workloads toward ready capacity without waiting for a root-cause diagnosis.

Three failure scenarios

NVIDIA walks through three cases to show how the loop behaves depending on the failure signal.

Hardware fault during training. When a GPU starts to fail, the symptom may first appear as ECC errors, a hung worker or a communication timeout. Recovery then splits into two parallel tracks. The job recovery service captures the job context and identifies the suspect node or GPU, and the scheduler integration restarts or reschedules the job. At the same time, the hardware recovery service isolates the node, records the reason and starts diagnostics. NVIDIA says the workload does not wait for root-cause analysis, and the faulty node is not handed to the next job.

Transient software fault. A training worker might exit, hit a recoverable timeout or briefly lose coordination while the node is healthy. If the failure history looks transient, the system retries according to scheduler policy instead of quarantining hardware that does not need repair. Retryable software failures restart automatically, and repeated failures are surfaced. NVIDIA says the benefit is restraint: no unnecessary operator noise and no shrinking of the scheduling pool.

Hardware degradation caught early. A node may show repeated GPU health warnings, connectivity problems or other risky signals. The hardware recovery service can remove it from scheduler eligibility proactively and start diagnostics before a training job lands there, so the job recovery service may never need to respond.

What it means for operators

For teams running AI workloads on Kubernetes with NVIDIA Run:ai, the update brings the same recovery framework that Slurm users already had. NVIDIA points readers to its documentation hub for upgrade instructions and to the full release notes, which cover further features including AI-enabled network anomaly detection. A solution overview explains how Mission Control is meant to simplify AI factory operations.