How Netflix keeps GPUs healthy for weeks-long training and inference jobs

Why GPU failures hurt so much

According to the Netflix Technology Blog, personalization on the home screen, from the rows shown to the artwork for each title, is chosen by machine learning models trained on how hundreds of millions of members watch. Those models are trained, and increasingly served live, on GPUs.

A single large training job can run for weeks across hundreds of GPUs on many machines. Netflix says such a job runs as a gang: the ranks move in lockstep and exchange data every step, so the job advances only as fast as its slowest participant and stops entirely if one participant dies. At this scale, individual GPUs and network links can fail during a run, so the platform has to handle failures without a person in the loop.

Netflix frames this as a business concern on two fronts. The first is availability, since a lost machine or a lack of healthy inference capacity delays work that members and creators depend on. The second is cost efficiency, since a GPU idling behind a slow neighbor, or a whole machine pulled because one device is degraded, is capacity paid for and not used. Inference is a growing share of the GPU footprint, so the same concerns increasingly apply to always-on serving.

Why basic health checks fall short

Netflix starts from the checks that its cloud provider runs on virtual machines: system and instance status checks, a storage check and scheduled maintenance events. These catch machines that are down, unreachable or slated for maintenance, but they stop at the instance level and do not verify that each GPU, its on-chip memory, its interconnect and its network fabric can support distributed training.

A GPU-level check that the driver responds and the GPUs are visible catches dead or missing devices, but Netflix lists cases it misses:

• A GPU can answer a status query and still be faulty once a workload touches it, for example through a fault in on-chip memory or device firmware.
• A network adapter may show fewer links than expected. Collective operations still run at a fraction of the bandwidth, so the job crawls instead of crashing.
• The checks themselves can fail. A diagnostic tool can hang when the driver hangs, and a binary can go missing after a driver upgrade so the check silently stops reporting.

The company argues that a check that can silently stop running is worse than none, because it gives false confidence. Its conclusion is that the health system must assume the hardware can lie and the tools that inspect it can break.

Four layers and how the checks work

Netflix treats reliability as four layers, each catching what the one below cannot:

• Node and hardware health: find bad GPUs, network adapters, interconnects, memory errors and thermal issues, then taint or recycle the machine.
• Scheduler and platform: retry at task and job level, validate a machine before it takes work, and drain and recycle machines safely.
• Training loop: checkpoint and use elastic workers so a job survives losing a machine.
• Observability: metrics, service-level objectives and failure attribution across the other three.

The first two layers keep hardware problems away from jobs, the third makes unavoidable loss cheap, and the fourth shows whether the rest are working.

On the checks, a node-level agent built on the open-source Kubernetes node-problem-detector runs on every GPU machine every 30 seconds. It turns results into conditions on the Kubernetes node, and controllers use those conditions to decide whether the machine can take work. GPU device checks use NVIDIA's DCGM for driver-reported error codes, uncorrectable memory errors, interconnect integrity and thermal state. Network checks inspect the count and link state of the Elastic Fabric Adapter interfaces that carry cross-machine traffic.

Netflix highlights four design choices:

• Failing closed: if the driver or diagnostic tool is unresponsive, the check times out and marks the machine unhealthy.
• A fallback path: if the tool that reads driver error codes is broken, Netflix reads the kernel log dmesg directly so no Xid or SXid errors are missed.
• Its own fatal-error map: vendor error catalogs lag newer GPU generations, so Netflix keeps its own mapping of which codes are fatal per GPU family. One generation emits a burst of normally fatal codes on every healthy boot, which Netflix now treats as noise.
• No silent no-ops: a check fails when a required dependency is missing. An upgrade once dropped a binary the network-adapter check relied on, and machines with the wrong number of adapters shipped for days before slow jobs were traced back to them.

What it means

The post, which is cut off before its section on automatic remediation, shows a platform built on the assumption that hardware will fail and that monitoring tools can fail too. For Netflix, the stated payoff is more useful work per GPU, which the company equates to spending less to do the same work. The details it shares on failing closed, fallback paths and per-generation error maps are practical lessons for other teams running large GPU fleets.