NVIDIA Open-Sources Attribution Tool to Trace Root Causes of AI Workload Failures

One fault, many symptoms

In distributed AI workloads, a single error can set off a cascade. NVIDIA says a GPU or fabric event may show up as NCCL errors on other ranks, scheduler state changes and thousands of downstream messages. The exception that gets reported shows where the workload stopped, not where the failure began.

NVIDIA Attribution is built to close that gap. It analyzes failures in the context of a workload rather than as isolated node events, and returns a structured diagnosis with the outcome, root cause, affected nodes, supporting symptoms and an evidence trail. The rules it uses are deterministic, which NVIDIA says makes results reproducible and customizable. The structured output is meant to feed existing health and recovery systems.

NVIDIA positions the tool as a complement to readiness checks, continuous health monitoring and the systems that repair infrastructure and restart workloads. It explains why a workload failed and provides evidence to guide the next action.

Two components and four stages

The release includes two deployable components, both built on a core attribution engine. The NVIDIA Attribution Service exposes attribution through a REST API. The NVIDIA Attribution Blueprint is a Python reference orchestrator that connects cluster schedulers and telemetry sources to the Service and writes the result. It ships with execution-context connectors for Slurm, Kueue and static inputs, plus log and metric connectors for Loki, local files, Kubernetes pod logs and PromQL metrics.

The Service analyzes one workload attempt at a time, in four stages:

• Scope the workload: the Blueprint identifies the attempt from scheduler identifiers, or from an explicit execution window and node allocation.
• Collect observations: connectors produce workload-scoped records from application logs, syslogs, GPU driver and Fabric Manager messages, scheduler history, and node and network metrics.
• Extract and correlate events: the engine identifies significant events such as GPU XIDs, NCCL failures, scheduler transitions and metric conditions, then applies deterministic rules to separate downstream symptoms from the cause.
• Return a structured result: the output includes the workload outcome (terminal_state), the attributed domain (sub_domain), symptoms, affected nodes, and a record of collector coverage and errors.

A worked example: NVLink signal integrity

NVIDIA's post walks through a synthetic multi-rank training workload. One GPU hits a hard fault, XID 149, and NCCL errors follow on other ranks. Application logs contain three different NCCL errors, while system logs on node-02 contain XID 149 with error words that decode to an NVLink signal integrity issue.

Extraction first turns the raw logs into typed events. An early filtering rule, HandleXidBurstEvents, decodes the error words and emits an attribution event for the signal integrity issue on node-02, leaving the three NCCL events unchanged. In final refinement, rules including Xid149AndNvlinkFiltering and Gb200PriorityRootCause keep the signal integrity symptom and demote the other node symptoms. Outcome mapping then yields a hardware-failure attribution.

The NCCL reports are not discarded. Demotion preserves their original sources as "interesting," so they can still be inspected. The workload is reported as a hardware failure rather than a set of independent communication errors.

NVIDIA also notes that attribution quality depends on the quality and completeness of the underlying telemetry. When multiple causes fit the evidence, sub_domain is marked ambiguous, so the diagnosis reflects uncertainty instead of turning incomplete evidence into a definitive answer.

Production use and what to do next

According to NVIDIA, the system analyzes millions of workloads each week across hundreds of thousands of GPUs. In one large RDMA-backed cluster, it correlated port errors, connection resets and scheduler events to identify patterns around specific host and switch ports, which helped teams target fabric repairs and congestion tuning.

For teams running their own clusters, NVIDIA points to three steps: clone the NVIDIA Attribution repository for the source code and sample workloads, follow the quickstart to run the sample workflow and inspect diagnoses in the local dashboard, and review the Blueprint configuration reference to connect their own scheduler and telemetry backends.