NVIDIA AI Cluster Runtime Targets Day-One Kubernetes Support for Vera Rubin NVL72

What AICR does

NVIDIA says configuring Kubernetes for new GPU hardware is complex and error-prone. Drivers, kernels, runtimes, networking, operators and schedulers all have to work together, and small mismatches can stop workloads from running or limit performance.

AICR is part of the NVIDIA DSX AI Factory Platform. According to NVIDIA, it captures tuned configurations in reusable recipes, generates bundles for existing deployment tools, and validates the running cluster. The company stresses that it is still early days for Vera Rubin, and the post covers what has been validated on early hardware access using the NVIDIA GPU Operator and the Dynamic Resource Allocation (DRA) driver with Vera Rubin support.

What changes for GB200 NVL72 operators

NVIDIA describes Vera Rubin NVL72 as an evolution of the same NVIDIA MGX rack-scale architecture used by GB200 NVL72, now in its third generation. Several user-visible pieces change:

• GPU driver: Vera Rubin NVL72 needs a new driver branch. It was validated with the host-installed R615.23 open GPU driver. Early support is targeted for R610, with official platform support in R620. AICR's GB200 baseline is R580.173.02, and NVIDIA says R615.23 is the tested version, not a confirmed minimum.
• Networking: ConnectX-7 NICs and BlueField-3 DPUs give way to ConnectX-9 SuperNICs and BlueField-4 DPUs. DOCA 3.3.0 is the first GA release supporting ConnectX-9.
• Device topology: Both reference trays have two CPUs and four GPUs, but the Vera Rubin NVL72 trays expose eight ConnectX-9 PCI device groups with 32 functions, versus four ConnectX-7 adapters on the DGX GB200 NVL72 compute tray. GPU-to-NIC affinity and NUMA placement should be checked for I/O-intensive workloads such as RDMA and GPUDirect Storage.
• IMEX: With DRA-managed IMEX, the host's static nvidia-imex service should be stopped, disabled and masked. DRA v0.5.0 adds an alpha host-managed option with domain isolation only, while the tested cluster uses driverManaged mode.
• MIG: Rubin GPUs introduce new MIG profiles, but the AICR recipes use whole-GPU allocation and MIG on Vera Rubin is not covered by this validation.

Validation results and early-silicon issues

NVIDIA deployed the AICR runtime bundle on Vera Rubin clusters using the official GPU Operator v26.7.0 release and DRA driver v0.5.0, with a host-installed R615 driver. Both the training and inference recipe variants now validate end to end for deployment and conformance. For training, the performance gate is a cross-node NCCL all-reduce over the NVLink fabric, run through an IMEX ComputeDomain, which reached 744.71 GB/s bus bandwidth at 16 GiB.

The process was not smooth. The first validation rounds were temporarily blocked by a hardware or fabric issue in one early cluster, which NVIDIA says was unrelated to the GPU Operator or the DRA driver and kept one compute tray out of the NVLink fabric. After the hardware was reseated, all GPUs on both nodes joined a single clean fabric clique. NVIDIA also reports that running AICR validation on early silicon exposed an unmapped Rubin architecture value in go-nvlib that crash-looped the DRA driver kubelet plugin. The fix propagated through the stack and shipped in the official DRA driver v0.5.0 release.

The Vera Rubin recipe family

The recipe family targets service rke2 and accelerator Vera Rubin NVL72 (vr200) on Ubuntu, with training, inference and NVIDIA Dynamo inference variants. It ships as Preview, which NVIDIA describes as an early-adopter path that does not yet carry the production support and lifecycle qualification of a Supported recipe while the platform is pre-release.

• The GPU Operator runs with driver.enabled: false, because the Vera Rubin system image pre-installs the arm64 open-kernel-module driver. The container toolkit stays operator-managed.
• IMEX-backed ComputeDomains are enabled through the DRA driver, while whole-GPU allocation stays on the device plugin.
• Kubernetes 1.34.1 or later is required: 1.34 for the GA resource.k8s.io/v1 DRA API, and the .1 patch because the RKE2 CDI/NRI toolkit path needs containerd 2.1.
• Ubuntu 26.04 with the Vera-optimized arm64 64k-page kernel is specified.
• The NFD topology updater's default 60Mi memory limit is raised, since it is OOM killed on multi-terabyte NVL72 workers.

What it means for platform teams

For teams running GB200 NVL72 today, the post gives a checklist of driver, networking, topology, IMEX and MIG differences to plan for ahead of upcoming releases. NVIDIA suggests generating the NVIDIA/aicr Vera Rubin training recipe with a single command, using the Dynamo platform flag for the inference variant, and reviewing the signed validation evidence for the training recipe. Because the recipes are Preview and the platform is pre-release, the results reflect early hardware rather than production qualification.