vLLM adds day-0 support for Recursive's Flint, hitting about 485 tokens/s on one GB200

A 40B MoE that activates 2.5B parameters

Flint is a 40B-parameter mixture-of-experts model that activates only about 2.5B parameters per token. It offers a 262K-token context and uses sliding-window attention in most layers. It ships as two separate checkpoints: NVFP4, which is quantization-aware trained, and BF16.

The architecture is a 49-layer pre-norm MoE with hidden size 2048. Of those layers, 35 attend to a 513-token sliding window with RoPE, while the other 14 attend to the full context with no positional encoding. Only those 14 keep KV for the whole sequence, so the cache costs 28 KiB per token. The model also uses learned attention sinks, GQA with 32 query and 4 KV heads, and per-head Q/K RMSNorm. It has 256 experts with sigmoid top-6 routing plus a shared expert, and layers 0 and 1 use a dense MLP instead.

The NVFP4 checkpoint is a GGUF file whose blocks use NVIDIA's NVFP4 format in a different byte order. vLLM repacks it losslessly at load time and runs it weight-only (W4A16), with BF16 activations and KV cache, as Recursive's reference implementation does. The vLLM team says every Flint kernel follows the reference's rounding points and that accuracy matches it on GSM8K, GPQA-Diamond, long-context MRCR and a tool-calling suite.

From 1,096 kernels to 346

The first implementation, built from vLLM's stock layers, was slow at small batch sizes. A decode step ran 1,096 kernels, many of them tuned for much larger batches. The team replaced them with Flint-specific kernels one change at a time, checking each against the reference. The listed gains are:

• Fused Q/K norm, RoPE and KV-cache write: 8% lower latency, one kernel instead of three.
• Small-batch MoE kernels: 27% lower, with router and expert GEMVs that read only the selected experts' weights. This was also the largest prefill gain.
• Small-batch dense GEMV: 25% lower, on the same packed NVFP4 weights as the large-batch kernel.
• Norm and SiLU kernels: 10% lower, following the reference's rounding.
• Decode attention with sinks and sliding windows, written in CuTe DSL: 5% lower.
• Low-latency decode path: 14% lower.

For up to 8 requests, the low-latency path runs each MoE layer as seven kernels. Q/K norm, RoPE, routing and the expert combine are fused into the neighboring GEMV kernels. Each kernel uses programmatic dependent launch, so it loads its weights while the previous one is still running.

In total, NVFP4 decode went from 4.81 to 1.78 ms (2.7×), BF16 decode from 3.53 to 2.34 ms (1.5×), and an 8K-token prefill from 207 to 145 ms (1.4×). A forward pass now runs 346 kernels instead of 1,096. Served end to end, batch-1 decode at 100K context takes 2.06 ms per token, about 485 tokens/s, which includes the LM head and sampling.

Running Flint on vLLM

The NVFP4 checkpoint is served with vllm serve RecursiveSI/flint-nvfp4-gguf, taking its config and tokenizer from the BF16 repo, RecursiveSI/flint. The BF16 checkpoint is served directly from RecursiveSI/flint. Both commands enable auto tool choice and use the glm47 tool-call and reasoning parsers, because Flint emits GLM-4.7-style tool calls and thinking blocks.

The flag --exclude-tools-when-tool-choice-none drops tool definitions from the prompt when tool_choice is "none", matching the reference. Default sampling settings are temperature 1.0, top-p 0.95 and top-k 20. Reasoning counts toward max_tokens, so the team advises allowing at least 32K for hard problems.

One limit applies: the Flint kernels target a single Blackwell GPU (TP=1, BF16 KV cache). Other configurations fall back to vLLM's standard layers, which, as the team notes of its first implementation, were slow at small batch sizes.

What it means for users

Teams that want to try Flint can serve either checkpoint with vLLM on one Blackwell GPU from the day of release, with low per-token latency for single requests, even at long context. Those running other hardware or parallelism setups will get vLLM's standard layers instead of the optimized Flint kernels.