vLLM Hits 954K Tokens/s per GPU on DeepSeek-V4.1-Flash with GB300 NVL72
The vLLM team reports that its engine, running with P/D disaggregation and Wide-EP on one NVIDIA GB300 NVL72 rack, serves DeepSeek-V4.1-Flash at 954K tokens/s per GPU on the SemiAnalysis AgentX benchmark. According to vLLM, its decode throughput stays within 6% of DeepInfer, DeepSeek's own engine, across batch sizes 8 to 256.
- vLLM reports 954K tokens/s per GPU on one GB300 NVL72 rack on SemiAnalysis AgentX, at a P90 of 196 tokens/s per user.
- That is 7.9× vLLM's single-node result from its first post; cost is $38.96 per GPU-hour at DeepSeek's off-peak prices.
- Decode throughput is within 6% of DeepInfer across batch sizes 8 to 256, per vLLM.
- vLLM does not claim a prefill speedup over DeepInfer because the test setups differ.
- A specialized decode path reaches 826 tokens/s for a single user.
Headline results on GB300 NVL72
In a post dated 23 October 2026, vLLM says that on a single NVIDIA GB300 NVL72 rack it reaches 954K tokens/s per GPU at a P90 of 196 tokens/s per user on SemiAnalysis AgentX. That is 7.9× the single-node result from the team's first post. The team also quotes a cost of $38.96 per GPU-hour at DeepSeek's off-peak prices.
A specialized decode path reaches 826 tokens/s for a single user, according to vLLM. Throughput in the benchmark counts cached input, uncached input and output tokens.
Comparison with DeepInfer
DeepInfer is DeepSeek's own inference engine for DeepSeek-V4.1-Flash. vLLM describes it as released the same week as an offline engine with published prefill and decode curves, and treats it as the most relevant reference for the model.
Decode. Under DeepSeek's decode configuration (131K context, nextn=3), vLLM on GB200 lands within 6% of DeepInfer's published results across batch sizes 8 to 256, according to the post.
Prefill. The setups differ. DeepInfer's published curve uses EP4DP4 with 32 redundant experts and a 140-token bounded replay, on hardware the chart does not state. vLLM's run uses DEP2 on GB300 with no redundant experts, a 128-token replay and its own prompts. For that reason vLLM says it does not read the gap as an engine-to-engine speedup, only that its DEP2 prefill tracks DeepInfer's curve across the published range.
The team attributes the closeness to a shared lineage: DeepInfer was originally built on vLLM, and vLLM has continuously integrated kernels from DeepSeek's libraries, including MegaAttn, MegaMoE and Mega-mHC. Unlike DeepInfer, which targets one model in offline mode, the same vLLM build also serves the model online across a full NVL72 rack, the post says.
What the model's design implies for serving
Before tuning, vLLM measured the model's architecture and drew several conclusions:
- The global KV cache takes only 890 bytes per token, thanks to cross-layer KV sharing and FP4 compression.
- Decode is MoE-bound even at long context. With DEP4 decoding at 128K context and 64 requests per rank, MoE takes 62% of the target step, while attention and the indexer together take 13%.
- Decode time barely grows with context. At DEP32 on GB300, going from 128K to 855K adds 0.6 ms (+7%) to target model decode latency, from 8.93 ms to 9.53 ms.
From this, the team concludes that prefill should run at a small DEP size, that KV capacity rarely limits batch size, and that Wide-EP is effective for decode because it cuts the dominant MoE weight-read cost. Decode load balancing can also be simplified to least-load by request count per rank.
Prefill optimizations and measurement method
In an AgentX prefill-only run at concurrency 256, DEP4 delivered 83% of DEP2's per-GPU throughput (53.3K vs. 64.4K tokens/s). Mirror and lag replays showed DEP4 losing 13.6% of throughput to DP imbalance, against 5.9% for DEP2, which makes DEP2 vLLM's default for prefill.
The post also lists fixes for bottlenecks that grow with context length. On two GB300 GPUs, an 8K-token prefill takes 87.9 ms with no prior context but 189.4 ms with 512K cached context. Extending SWA bounded replay to layer 20 (#59894) improves prefill throughput by 19.9% at 32K new tokens, with gains from 10.1% at 4K to 23.1% at 64K. Using 2 MB pages for shared Engram host tables (#62359) adds another 2.8% at 32K, by avoiding frequent GPU TLB misses.
For measurement, vLLM uses vllm bench forward (#61339) to time exact prefill or decode batches, and DecodeBenchConnector for end-to-end decode at a given concurrency.
What it means
For operators of DeepSeek-V4.1-Flash, the results, which are vLLM's own, suggest a general-purpose open serving engine can come close to a model-specific one while also handling online serving at rack scale. The post says the gains come from engineering on components vLLM already has, built on kernels DeepSeek open-sourced. The source text provided is cut off partway through the optimization list, so further details are not covered here.
