vLLM Adds Support for NVIDIA Vera Rubin NVL72, Cites 7.8x Throughput Gain
The open-source inference engine vLLM now runs on NVIDIA Vera Rubin NVL72, with daily container builds and support for models from DeepSeek, Moonshot AI, Z.ai and MiniMax. In an early look at the work, the vLLM team reports 7.8x the throughput per GPU versus GB200 NVL72 on AgentX at matched interactivity.
- vLLM now runs on NVIDIA Vera Rubin NVL72 with daily container builds, per the vLLM project.
- Early results: 7.8x throughput per GPU versus GB200 NVL72 on AgentX at matched interactivity.
- Up to 3.7x higher VLM throughput versus GB300 NVL72 in MLPerf.
- Blackwell kernels (sm100f) run on Rubin (sm107), and models from DeepSeek, Moonshot AI, Z.ai and MiniMax are supported.
- CUDA 13.4 locality domains gave an average 1.2x speedup in a preliminary MoE decode test.
What vLLM says it has achieved so far
According to the vLLM project, engineers from Inferact, NVIDIA, Red Hat and the wider community have been bringing vLLM up on Vera Rubin NVL72 since the platform was announced. The post describes the state of the work as an early look, not a final result.
The headline numbers are 7.8x the throughput per GPU versus GB200 NVL72 on AgentX at matched interactivity, and up to 3.7x higher VLM throughput versus GB300 NVL72 in MLPerf. The team says it expects more performance as optimizations continue.
Model coverage already includes DeepSeek, Kimi, GLM and MiniMax models, according to the post.
What Rubin changes for inference
vLLM describes Vera Rubin as a platform built for agentic inference, made up of five rack-scale systems: Vera Rubin NVL72, Vera CPU rack, Groq 3 LPX, Spectrum-6 SPX and BlueField-4 STX Storage. Compared with GB200 NVL72, the post lists these per-system differences:
- Compute: 5x more NVFP4 inference FLOPS.
- Memory: HBM upgraded from HBM3e to HBM4, with up to 2.4x higher bandwidth.
- Networking: sixth-generation NVLink with up to 1.7x more bandwidth, which the post says should speed up collectives such as AllReduce and All2all and help prefill/decode disaggregation and wide expert parallelism.
- Softmax: higher exponential throughput, including 2x FP32 and 4x BF16/FP16 versus GB200, so softmax keeps pace with faster matrix operations.
Software: Blackwell compatibility and locality domains
Rubin builds on Blackwell's architecture family. It is a new GPU compile target (sm107), but kernels built for the Blackwell family target (sm100f) can also run on it. In practice, vLLM says its Blackwell kernels, especially GEMM-heavy ones such as attention and MoE, already run on Rubin without modification.
Through FlashInfer 0.7.0, vLLM also gets Rubin-tuned attention, GEMM and MoE kernels, and the team has tuned its MiniMax Sparse Attention (MSA) prefill kernel for Rubin.
A further change uses the locality domain feature in CUDA 13.4. It lets applications place computation and data in the same domain, so streaming multiprocessors (SMs) read from nearby memory with higher bandwidth and lower latency. vLLM applies this to MoE decode, which is bound by reading weights from HBM. It splits the weights column-wise across two domains and restricts each domain's SMs to their local shard.
In a preliminary test using MiniMax M3 MoE shapes, the team reports an average 1.2x speedup in small-token forward settings, with a similar trend across other TP and EP strategies. Using backfill mode engages all 212 SMs, while default mode uses 200 and still gives a similar gain. The post stresses these are early results with room for tuning, and that locality domains remain in active design and development.
What it means for users
Users can already try the setup. The post says nightly images built with CUDA 13.4 and PyTorch 2.15 are available on Docker Hub as vllm/vllm-openai:cu134-nightly for Rubin hardware. Because Blackwell kernels carry over, teams running vLLM on Blackwell should not need to start from scratch, according to the project. The performance figures are the project's own early measurements, and further optimization is expected.
