NVIDIA Introduces Open Source Tools for Kubernetes GPU Cluster Operations

A lifecycle approach to GPU cluster management

NVIDIA DSX OS is part of the NVIDIA DSX AI Factory Platform, which brings together technologies for designing, building and operating AI factories. The software layer addresses a problem that can emerge after a cluster passes its initial readiness checks: nodes can be replaced, drivers can be patched and capacity can be added, changing the cluster’s condition over time.

According to NVIDIA, one underperforming GPU can slow an entire distributed training job even when basic health checks and utilization metrics appear normal. The company points to goodput, the volume of work successfully completed, as a more useful measure of cluster efficiency than utilization alone.

DSX OS is organized around three open source projects, all licensed under Apache 2.0 and open to external contributions: NVIDIA AI Cluster Runtime, NVIDIA Cluster Readiness Engine and NVSentinel.

AICR creates a reproducible software baseline

NVIDIA AI Cluster Runtime, or AICR, defines version-locked configuration recipes for GPU-optimized Kubernetes infrastructure. The recipes can cover components such as networking, storage and drivers, while dependency graphs specify which components must be installed and healthy first.

AICR renders those recipes as deployment artifacts for existing tools, including Argo CD, Flux, Helm, Argo CD with Helm and Helmfile. This allows operators to retain their current deployment tooling while using AICR to generate configuration bundles. The recipe files can also be managed through version history, code review and drift detection.

The source describes health assertions for individual components as another part of the baseline. These checks can identify conditions that a pod-phase check might miss, such as a container stuck with a waiting reason associated with CrashLoopBackOff.

NVCRE tests production readiness

NVIDIA Cluster Readiness Engine, or NVCRE, is designed to determine whether a cluster can run production workloads, rather than merely confirming that basic services are running. It tests GPUs, networking and storage using AI workloads, diagnostics and NVIDIA Collective Communication Library, or NCCL, tests.

The validation process can run training jobs and diagnostic tests across multinode groups. When a group fails, NVCRE can bisect it to identify suspect nodes, then record pass or fail results, failing node IDs and failure reasons in a Certification resource.

NVSentinel manages validation requests, scheduling gates, quarantine and repair, while NVCRE runs the tests and records their results. Nodes can be cordoned or tainted during validation so workloads are not scheduled on them. The validation outcome determines whether that scheduling gate can be removed.

NVSentinel monitors health and supports revalidation

NVSentinel continuously monitors GPU, node and network health signals and publishes node conditions. When configured, it can cordon or taint unhealthy nodes, drain workloads and remediate faults. It can also trigger NVCRE revalidation before repaired nodes return to service.

The walkthrough published by NVIDIA follows a cluster from configuration through certification, operation and recertification after repair. AICR drift detection can also trigger certification runs after infrastructure changes, including driver upgrades. For users operating Kubernetes GPU clusters, the tools provide an open source way to connect deployment consistency with workload-level validation and ongoing node health management.