vLLM Adds Day-0 Support for Moonshot AI’s Kimi K3.1 Pro

Sparse attention targets long-context costs

Kimi K3.1 Pro keeps Kimi K3’s hybrid design, with 69 KDA layers and 24 full-attention layers, but adds a Distributed Sparse Attention indexer to all 24 full-attention layers. According to vLLM, the indexer scores earlier tokens and selects the top 2,048 before MLA attention runs over those tokens’ latent representations.

The change is designed to reduce memory traffic during long-context decoding. vLLM says dense MLA reads 576 bytes of FP8 latent data per context token and layer, while the sparse design scans 132-byte FP8 index keys and then reads the full latent for only 2,048 tokens. At long context, that represents about 23% of the bytes read by dense MLA per layer, according to the company.

Kimi K3.1 Pro has 2.8T total parameters, with 104B active, and a 1M-token context window. Its 896-expert MoE architecture activates 16 experts, while the model’s full-attention layers use 32 heads with 128 dimensions and FP8 keys.

Packed caching and parallel serving

vLLM combines the model’s MLA latent, indexer keys and KDA state in one block-based cache. The company says this layout allows FlashInfer and FlashMLA sparse kernels to use the cache with FP8 or NVFP4 key-value data, plus optional MXFP4 index keys. It also lets disaggregated serving transfer each block with one large RDMA read instead of multiple per-layer reads.

On 8× GB300, vLLM reports that token-sharded prefill reached a 1.69× time-to-first-token improvement over head-sharded prefill at 340K prompt tokens. Its decode context parallelism approach uses an exact top-k merge with about 10× less traffic, making the attention module 1.3, 1.5× faster at DCP8, according to the company.

The same tests reported 1.2, 1.6× faster decode on 8× GB300 through a fused decode path, and 1.9× faster decode under DCP8. With NVFP4 latent data and MXFP4 index keys, vLLM says eight GB300 GPUs can hold 1.7× as many tokens as with FP8, while DCP8 can raise capacity to about 36 full 1M-token contexts, compared with 3 for TP8 with FP8.

Support includes agentic serving and reinforcement learning

vLLM also reports agentic-serving results for Kimi K3.1 Pro. On NVIDIA GB300, the model reached up to 375.9 tok/s per user at the latency-sensitive end and 37.8K total tok/s per GPU at the throughput end. On NVIDIA Rubin, the figures reached up to 815.6 tok/s per user and 68.7K total tok/s per GPU on the SemiAnalysis AgentX benchmark.

The release includes what vLLM describes as zero-mismatch RL support on day 0. The company says vLLM and Megatron produce bit-identical log-probabilities for the base model and LoRA adapter, resulting in a PPO KL of exactly 0 at every step of a 100-step full-model run.

For users, the announcement provides serving recipes for NVIDIA and AMD GPUs, including commands for tensor parallelism, fast safetensors loading, tool calling, reasoning and speculative decoding. The reported results are vLLM’s measurements and describe the infrastructure changes it says are intended to make Kimi K3.1 Pro more practical for long-context, agentic and reinforcement-learning workloads.