vLLM Gets DeepSeek-V4.1-Flash to 5.3x Throughput in Three Weeks

A model built for long-horizon agent serving

According to vLLM, DeepSeek V4.1 uses a causal encoder-decoder (CED) architecture aimed at long-horizon agentic serving. The model activates 16B parameters per token during decode but only 8B parameters during prefill.

It is also frugal with memory. The model combines Compressed Sparse Attention 2 (CSA2), an FP4 KV cache and inter-layer KV cache sharing, which brings the global KV footprint to 890 bytes per token. vLLM says it paired these model-level features with its own system optimizations to reach the 5x gain on the SemiAnalysis AgentX agentic serving benchmark.

SWA bounded replay cuts redundant prefill work

DeepSeek-V4.1-Flash keeps two kinds of KV cache. The global KV is compressed, shared across layers and stored in FP4. The sliding-window (SWA) KV is uncompressed in FP8 and covers the last 128 positions in each of the 40 layers. According to vLLM, SWA KV is costly in two ways: prefix caching must store it at every possible hit boundary, at more than 10x the storage of global KV, and prefill runs layers 21 to 39 on every prompt token even though decode only reads their last 128 positions.

Exact recomputation would be expensive, since rebuilding a window across L layers means replaying roughly L x 128 tokens. SWA bounded replay trades exactness for speed: it reruns only the last 128 tokens and clips the SWA window at the replay start. vLLM applies it in two places:

• Encoder side: vLLM caches only the global KV. On a prefix hit of length H, it reruns tokens [H - 128, H) to rebuild the SWA KV.
• Decoder side: layer 20 computes the decoder's global KV and layers 21 to 39 reuse it. vLLM runs layer 20 on every token and layers 21 to 39 only on each request's last 128 tokens, which for long prompts skips nearly half the model.

Because the trimmed layers do so little GPU work, launching their kernels eagerly leaves the GPU idle. vLLM therefore captures layers 0 to 20 on the full batch and layers 21 to 39 separately on the trimmed batch, using its breakable PIECEWISE CUDA graph. The feature is on by default for DeepSeek-V4.1 and is controlled by the --[no-]swa-bounded-replay flag.

Accuracy and speed results

The method is not bit-exact, but DeepSeek reports negligible quality loss. vLLM says it confirmed this on benchmarks including GSM8K and GPQA, with gaps within about 1.5 standard errors.

On speed, vLLM reports that decoder replay with CUDA graphs cuts prefill computation time by 30 to 40%, and the post's summary cites a roughly 30% reduction in time to first token. CUDA graphs matter most for short prompts. Without them, launch overhead outweighs the GPU savings and replay can be slower than the baseline, by up to 12% at 1K on DEP2. For long prompts, eager replay already captures most of the gain.

New kernels from DeepSeek and vLLM

DeepSeek released the model together with new kernels in three repositories: DeepSelect, a top-k library for DeepSeek Sparse Attention; sparse indexer and fused GEMM kernels in DeepGEMM; and NVFP4 KV cache support plus the fused MegaAttention kernel in FlashMLA. vLLM has integrated several of them. Its reported figures include:

• Mega-mHC: fuses the post step, the delayed-pre step and RMSNorm, and is 1.14 to 1.51x faster than the TileLang version on NVIDIA GB200.
• Mega-Gate: fuses the MoE router into one kernel, with 1.18 to 1.31x kernel speedups at medium batch sizes.
• mHC multistream overlap: computes the next block's coefficients on a side CUDA stream, reducing latency by about 4% in the TP4 low-latency scenario.
• Sparse MQA logits: scores only candidate positions. Per layer on NVIDIA GB300 it is 1.2x faster at 8K tokens and 14 to 23x faster at 512K. End to end on 4x GB300, decode improves 3 to 6%, and prefill is 1.43x faster at 512K and 2x faster at 1M context.
• MegaAttention: combines query RoPE, sparse attention, inverse RoPE and the FP8 cast in one launch, and reads an NVFP4 compressed KV format that is 45% smaller than the previous FP8 KV cache. vLLM cites a 1.45x kernel efficiency gain.
• Low-latency fused WO-A kernel: improves the fused path by up to about 2.1x and lowers inter-token latency by up to about 6 to 7% at low concurrency.

vLLM also describes work on the model's Engram layers, which look up rows from two large FP8 tables. It prefetches CPU-offloaded lookups asynchronously, shards Engram heads with a unified TP/DP scheme, and supports transparent huge pages, which it says gives up to 10x faster lookup kernels for prefills.

What it means for users

For teams serving DeepSeek-V4.1-Flash with vLLM, the reported gains apply to both latency-sensitive and throughput-bound workloads, and SWA bounded replay is enabled by default. The numbers are vLLM's own and were measured on specific hardware such as NVIDIA GB200 and GB300 and on the AgentX benchmark, so results on other setups may differ. The post also shows how quickly an inference engine can close the gap after a model launch when the model developer ships optimized kernels alongside the weights.