Together AI Details Auto Repair for GPU Clusters, From Fault to Healthy Node

Why NotReady is not enough

Together AI argues that at cluster scale, GPU faults are routine. It lists an Xid error, a double-bit ECC error, a GPU falling off the PCIe bus and a flapping InfiniBand link as examples that can stall a distributed training job or take an inference replica down. Spotting a problem is usually the easy part, the company says. The hard part is everything between the fault and a healthy node.

The company says most node auto-repair keys on a single signal: the node went NotReady. That signal looks the same whether a node is broken or briefly unreachable, and it does not say why the node failed, so there is no basis for choosing between a reboot and a new machine. When a storage or network event touches many nodes at once, every affected node drops to NotReady. Together AI says its system has a built-in circuit breaker to prevent mass replacement of nodes.

Its rule is simple: no node repair occurs without a failed health check behind it, and every repair carries its supporting evidence. According to Together AI, its GPU cluster customers and its internal inference platform already use the capability, which the company announced earlier this year.

Detection on both sides of the hypervisor

GPUs in Together GPU Clusters are passed straight through to the customer's VM for bare-metal performance. GPU faults are therefore visible from inside the VM, while host and fabric faults are visible from the host. Detection runs on both sides and lands in one place.

• Inside the VM, exporters watch the kernel log for Xid errors and GPUs falling off the PCIe bus.
• DCGM covers ECC double-bit errors, row-remap failures, PCIe replay rate and thermal throttling.
• Node Problem Detector covers read-only filesystems, XFS shutdowns, kernel deadlocks and repeated kubelet or containerd restarts.
• On Slurm, an exporter reports nodes marked DOWN or DRAIN.
• On the host, Together AI watches InfiniBand rail state and link flapping, attributed to the VM that owns those rails, plus host reachability.

Every alert carries a stable summary, a plain-language description and a machine-readable evidence payload. For an Xid, that is each code and its NVIDIA mnemonic. For thermal throttling, it is the peak temperature and throttle count. The company keeps observed and actionable signals separate: every repair-relevant alert marks the node Degraded, but only signals with an action mapping create a repair. Disk pressure, for example, is shown but does not trigger a reboot, because a reboot would not fix it. Alerts without a mapping are counted, logged and routed to Together AI's support team.

Customers set the policy

Together AI says the platform decides what action fits each signal, mapping it to the lightest fix, while the customer decides whether and when to act.

In the Repairs tab, the Auto-remediation policy defaults to Approve before repair, so every recommendation waits for a person who can accept or override it. Customers can switch to Fully automatic, but not as a blanket setting. A checklist groups faults by the repair they trigger, and consent can be set per Xid code. One common setup, the company says, auto-approves reboots for Xid 119 and 95 while keeping migrations gated.

Timing is also configurable. Because a repair drains the node, Wait for idle holds it until the node is idle. On Kubernetes that means no running pod requests a GPU, and on Slurm it means no allocation on the node. Maximum wait defaults to 2 hours with a cap of 24, and Do not interrupt running jobs removes the bound entirely. Together AI advises setting the wait to at least the checkpoint interval. Each repair records who requested it, who approved it, which rule decided and when.

Bring your own detector and the escalation ladder

Teams that already trust tools such as DCGM dashboards, NVSentinel, in-house scanners or job-level checks can feed repairs into the same pipeline. There are three entry points: the API, the console, and node labels, which are in preview. With node labels, any detector with RBAC permission to write node metadata can open a repair by adding one annotation and one label. The repair is recorded with the detector as an external requester. Detector name plus request ID acts as an idempotency key, so a detector that re-asserts the same fault cannot reboot a node twice. Annotation requests can ask for Reboot, Quick reprovision or Migrate, while Remove stays human-only. They still pass the evidence gate, capacity check and budgets, and a per-cluster cap limits damage from a buggy detector.

Every repair follows the lifecycle of cordon, wait, drain, act and rejoin. The actions run from lightest to heaviest:

• Reboot: restarts the VM in place, and local /scratch and /tmp data survive.
• Quick reprovision: rebuilds the VM from a clean image on a healthy host, and local data is lost.
• Migrate to new host: rebuilds the VM on different hardware and sends the old host to re-validation.
• Remove: pulls the node for RMA and is always human-initiated.

If a fault returns on the same machine after the same action has run twice, the next recommendation moves up one rung. A third round of Xid reboots becomes a reprovision, and a third reprovision becomes a migration. The ladder is tracked per machine, cluster, alert and Xid code.

Availability and what it means for users

According to Together AI, passive health checks, repair recommendations, the auto-remediation policy, wait-for-idle controls, the escalation ladder, the safety rails and API and console repair are generally available. Node-label based remediation contracts are in preview. The company says it is continuing to add detection signals and remediation actions.

For teams running large GPU fleets, the design keeps humans in control of when and whether repairs run, while tying each action to evidence and an audit record.