NVIDIA Snapshot restores GPU workloads in seconds on Kubernetes

Snapshot preserves initialized GPU workloads

Snapshot is a new open-source component of the NVIDIA DSX AI Factory Platform. It captures the CPU and GPU memory of a fully initialized workload as a portable checkpoint artifact, then restores that state on compatible infrastructure.

The system is designed to avoid repeating several costly startup steps, including loading model weights into GPU memory, compiling kernels and capturing CUDA graphs. NVIDIA says infrastructure provisioning and container image loading remain outside the measured restore path.

Snapshot provides two operations for GPU Pods. A checkpoint captures a running workload, while a restore starts a new Pod from the saved artifact on a node with the same GPU model and driver. Both operations use Kubernetes custom resources, and the workload needs a small handshake to pause before checkpointing and resume after restoration.

Benchmarks show restores under 43 seconds

NVIDIA tested Snapshot with vLLM, SGLang and NVIDIA TensorRT LLM on NVIDIA B200 GPUs, using VAST NFS storage. The tested models ranged from 600 million parameters to mixture-of-experts models with checkpoints of up to 1 TB across eight GPUs.

Across all tested configurations, Snapshot restored replicas in 3.6 to 42.7 seconds, which NVIDIA reports as 8x to 36x faster than cold starts. The largest mixture-of-experts models, with checkpoints of about 600 GB to 1 TB across eight GPUs, restored in under 43 seconds, compared with cold-start times of 6 to 19 minutes.

Checkpoint creation took 15 seconds to 5.5 minutes, depending on the model and configuration. NVIDIA says the combined checkpoint and restore time was shorter than a single cold start in every tested configuration. In the largest eight-GPU workloads, restoring GPU state took 18 to 29 seconds, while CRIU-based CPU process restoration took 4.2 to 10.8 seconds.

Support for multi-GPU inference and integrations

Snapshot can checkpoint and restore compatible workloads spanning multiple GPUs on a single node as one artifact across ranks. NVIDIA says this supports large mixture-of-experts models using tensor parallelism, data and expert parallelism, and decode context parallelism.

The repository includes examples for vLLM, SGLang and NVIDIA TensorRT LLM. Although Snapshot was originally developed for NVIDIA Dynamo inference workers, NVIDIA says other inference engines, schedulers, controllers and platforms can integrate the checkpoint and restore capabilities.

For users and platform teams, the approach is intended to make initialized workloads reusable when inference demand increases or a replica fails. The benchmark results indicate that restoring saved state can reduce the delay before serving resumes, while compatible GPU, driver and runtime requirements still determine where a workload can be restored.