vLLM adds day-0 support for Mistral Large 4, a 1-trillion-parameter MoE model
The vLLM inference project now supports Mistral Large 4 from day 0. According to vLLM, the model is Mistral's new frontier mixture-of-experts, natively multimodal model, and serving it required new work on attention, parallelism and kernels rather than just a model file.
- vLLM supports Mistral Large 4 from day 0: 1T parameters total, 49B active per token, multimodal with a Pixtral encoder.
- FP8 and NVFP4 checkpoints are available; vLLM says NVFP4 stays within 2.1 points of FP8 on evals while decoding 1.08 to 1.19x faster at TP8.
- The built-in DSpark drafter gives about 2x output throughput on TP8, according to vLLM.
- DCP raises KV capacity from 2.38M to 18.88M tokens (FP8) on TP8, and PCP+DCP cuts TTFT by up to 1.6x.
- AMD Instinct MI350 Series GPUs are supported on day 0 with ROCm.
What Mistral Large 4 is
According to the vLLM team, Mistral Large 4 has 1 trillion parameters in total and 49 billion active per token (52 billion including embeddings and output layers). It is a 72-layer mixture-of-experts model with 128 routed experts (top-4) plus 1 shared expert and a hidden size of 6144.
Its attention stack is DeepSeek-style and sparse: MLA (KV LoRA rank 512, RoPE dim 64) combined with a DSA indexer that selects the top 2048 keys per query. On top of that, vLLM lists several new architecture features: cross-layer KV sharing, NoPE layers, SSMax and LeRoPE. The model ships with a built-in DSpark speculative decoding drafter and a Pixtral vision encoder, which handles image inputs, including multiple images per prompt.
Two checkpoints are available: FP8 with block-scaled weights, and NVFP4, which uses NVFP4 experts and FP8 attention and shared experts. Both use an FP8 latent KV cache (fp8_ds_mla) by default.
How to run it and what vLLM reports
Support ships in the vllm/vllm-openai:mistral-large-4 Docker image, which also includes mistral-common 1.13. The FP8 model is served on 8 GPUs with tensor parallelism of 8. The NVFP4 model runs on 4 Blackwell GPUs, or on 8 with --tensor-parallel-size 8. Both examples enable the DSpark speculative config with 3 speculative tokens, the Mistral tool-call parser and the Mistral reasoning parser. For multi-node and disaggregated setups, vLLM points to a Mistral Large 4 recipe.
vLLM says the DSpark drafter (3 sliding-window MLA layers, depth 3) accepts about 2.9 tokens per step and gives about 2x output throughput on TP8.
On NVFP4, vLLM states that it does not trade quality for speed. Across multimodal, long-context, agentic coding, business automation and cyber evaluations, it stays within 2.1 points of FP8 on every task, with a mean difference of -0.3 points. Mean completion tokens stay within about 4% of FP8. At TP8, NVFP4 decodes 1.08 to 1.19x faster per token than FP8 without speculative decoding (1.18 to 1.22x with DSpark) and holds about 1.5x the KV cache.
- Benchmarks named include GDP.pdf, MathVision, Graphwalks 512k, DeepSWE, SWE-bench Pro, Terminal-Bench 4, AutomationBench and CyberGym.
- AMD Instinct MI350 Series GPUs are supported on day 0 with ROCm, including DSpark, image inputs and expert parallelism with MXFP4 experts.
Engineering work behind the support
Mistral Large 4 runs only on vLLM's model runner V2 (MRV2) and uses breakable CUDA graphs without torch.compile. vLLM says the graphs gave about a 6x speedup over eager mode. Hand-written fusions, such as SSMax with MLA RoPE and the indexer query and key paths, cut decode TPOT by 8 to 10%. A later round, including an FP32 router GEMM and a fused all-reduce, residual and RMSNorm kernel, lowered c1 TPOT on TP8 FP8 from 10.22 to 9.00 ms.
The model code lives in a new vllm/models/mistral/ package, with per-layer features driven by a layer spec built from params.json rather than model-specific branches in shared code. Cross-layer KV sharing was added through a kv_sharing_target_layer_name field. Nine NoPE layers read the latent cache of layer 35, so 63 of the 72 layers own a cache. vLLM describes this as model-agnostic and is upstreaming it in vllm#64095. Per token, the KV cache stores 47,136 bytes: 41,328 for the MLA latent and 5,808 for the indexer.
Long context and disaggregated serving
Because MLA keeps one latent KV head, the cache is replicated on every rank under tensor parallelism. Decode context parallelism (DCP) shards it by token instead. On TP8, vLLM reports that DCP8 raises KV capacity from 2.38M to 18.88M tokens for FP8 and from 3.60M to 28.76M tokens for NVFP4, about 7.9x more. It is enabled with --decode-context-parallel-size 8 and requires the default FP8 KV cache. Prefill context parallelism combined with DCP cuts time to first token by up to 1.6x, and DSpark works under both.
vLLM says DCP with sparse MLA initially cost about 7 ms per decode step at 60k context. It replaced NCCL all-gathers and reduce-scatters with a symmetric-memory all-to-all combine (vllm#61285), a direct top-k exchange between ranks and top-k compaction for batches of 32 or more tokens.
The release also covers NIXL-based prefill/decode disaggregation with DP/EP, TP and PCP×DCP prefill nodes, with DSpark on both sides, and HiSparse host-memory sparse KV offload, now supporting cross-layer KV sharing.
What it means for users
For teams that self-host models, the release gives a ready Docker image and tested commands for running a 1-trillion-parameter model on FP8 or NVFP4 hardware, including AMD GPUs. The NVFP4 results reported by vLLM suggest a lighter option with similar eval scores, while DCP targets workloads that need very long contexts. Several of the changes, such as KV sharing and the context parallelism improvements, are described as generic, so other models may benefit as they are upstreamed.
vLLM: Day 0 support for Mistral Large 4 on vLLM
