NVIDIA Recovery automates restarts for failed AI training jobs

Why manual recovery does not scale

NVIDIA describes the problem in terms of goodput, the share of wall-clock time that turns into productive training progress. A stalled training rank can trigger an NVIDIA Collective Communications Library (NCCL) timeout, a GPU node can stop responding to the scheduler, or a straggler can slow an entire job. Each case is manageable alone, but the cumulative recovery overhead adds up.

Manual handling requires log analysis, fault isolation and a restart from a saved checkpoint, while affected GPUs sit idle. NVIDIA argues that as jobs grow in number and size, overlapping failures raise the engineering burden and the wasted GPU-hours, which increases compute costs and extends time to train.

The company gives an example of scale: if each GPU runs a day without failure with 99.9% probability, a job spanning 10,000 GPUs completes that day uninterrupted with a probability of only about 0.005%. NVIDIA also cites an 11-month study of 24,000 A100 GPUs at Meta, in which hardware failures interrupted fewer than 1% of jobs, but those jobs accounted for 19% of GPU runtime.

The four-stage recovery loop

NVIDIA Recovery detects interruptions using application logs, GPU metrics and scheduler signals. Four stages organize the loop, with actions depending on the interruption and the configured policy:

• Detect: an Anomaly Detector combines application logs and scheduler signals to identify crashes, hangs and stragglers. It recognizes PyTorch and NCCL failures, CUDA errors and interconnect failure patterns.
• Attribute: the service classifies the interruption using the gathered evidence. Optional integration with NVIDIA Attribution adds deeper diagnosis to distinguish hardware faults, system-software issues and application bugs.
• Isolate: implicated nodes are excluded from further use.
• Restart or stop: workloads are restarted or stopped according to policy.

Missing progress signals or recognized communication-hang patterns indicate potential hangs, while sustained throughput degradation against a recent baseline indicates potential stragglers.

A persistent audit trail records every detection pattern, action taken, timestamp, workload identifier and attribution verdict. A Recovery Console provides a live operator dashboard across scheduler types.

Where it fits in the NVIDIA stack and with Slurm

NVIDIA positions Recovery as a dedicated workload recovery layer within the DSX OS stack. Node health monitoring and isolation depend on the cluster's orchestration. Kubernetes deployments use NVSentinel to monitor GPU health metrics and XID events and to cordon and drain faulty nodes. Many Slurm sites check node health with custom prolog and epilog scripts. Inside individual jobs, the NVRx library handles in-job recovery and hang detection.

NVIDIA Recovery works at the cluster level, managing job restarts, placement and node exclusion across workloads. The layers operate independently without hard dependencies, so sites can adopt components separately or plug in their own. According to NVIDIA, a cluster already using NVSentinel or NVIDIA Health Validation can integrate Recovery with little or no change to existing health workflows, and for NVRx users it covers the cross-workload lifecycle that in-job restart cannot reach.

For Slurm, the integration uses a reference Advisor Controller that can be deployed through a Helm chart in either in-cluster or SSH mode. This keeps the recovery service independent of scheduler internals.

Production numbers and getting started

NVIDIA reports that from July 9 to Oct. 9, monitoring on one training cluster covered nearly 200 jobs per day and recorded approximately 8,550 anomaly detections. Mean detection latency was 69 seconds and mean recovery-action latency was 94 seconds. The company notes that recovery-action latency ends at the automated action, so scheduling and checkpoint restoration determine when productive training actually resumes.

New deployments start in monitor mode by default. In this mode the service publishes recovery recommendations without taking scheduler actions, so operators can check signal quality before enabling autonomous recovery.

NVIDIA points users to the dsx-ai-factory/NVIDIA-Recovery repository for source code and the issue tracker. A quickstart guide allows local deployment without a physical GPU, and an operator deployment guide covers in-cluster and SSH-based Slurm setups.

What it means for operators

For teams running many training jobs, the pitch is less dependence on individual engineers and per-user recovery arrangements as interruptions accumulate. The monitor-first default lets operators test the service's judgments against their own clusters before handing over control. NVIDIA says the approach suits both small and large distributed runs: many small jobs multiply the operational burden, while in large jobs each interruption affects more GPUs.