vLLM Speeds Up Kimi K3 Decoding on GB300 With Agent-Written Kernels

What vLLM built for Kimi K3

According to vLLM, the new path uses agent-assisted development to create a unified CuTe DSL decode implementation covering attention, MoE and communication across 8 or 16 GPUs. It replaces and fuses almost all operations with customized CuTe DSL kernels tuned for specific subproblems.

The team reports 1.5-2× higher decode throughput in several tested configurations and up to 1.8× higher P90 interactivity on the AgentX workload. A kernel timeline in the post compares stock vLLM, which uses 130 microseconds and 23 kernels per layer, with the specialized decode path at 69 microseconds and 13 kernels.

The path targets NVIDIA Blackwell GPUs with 8 or 16 GPUs, decode batches from 1 to 64, and optional decode context parallelism for long contexts. The GPUs must share one NVLink domain with multicast support.

Four optimization techniques

vLLM describes four main areas of work.

• Shape tuning: small-matrix kernels are used for attention input projections at batch sizes 1-2, and split-K kernels at larger batches. KDA gets its own CuTe DSL decode kernel, and decode context parallelism splits the KV cache across GPUs for long contexts.
• Overlapping compute and communication: the attention output projection is fused with its all-reduce and the following attention-residual computation. At larger batches, a two-stage reduction reduces data movement between GPUs.
• MoE scheduling: router scores and latent input projection, which the baseline computes in full on every GPU, are split across TP ranks and combined into one matrix multiplication. The shared expert runs on a separate stream.
• Weight prefetching: in small-batch inference, which is often memory-bound, a dedicated kernel on a side stream loads upcoming weights into cache while communication runs, raising cache hit rates.

A fast-track model for agent-written code

The post argues that agent-optimized inference code is outpacing the team's review and maintenance workflow. Understanding every line of every kernel and testing every supported configuration is very time-consuming, but skipping that work risks shipping buggy software that fails in production. vLLM points to a proposed fast-track model optimization RFC as the foundation for its answer, with the stated aim of decreasing review burden while managing risk.

The approach rests on isolating changes within the model's code, bounding the scope, validating correctness systematically, and scaling validation by letting contributors run standard evaluations on their own GPUs and share reproducible results before requesting maintainer attention. Kimi K3 is the first contribution made this way.

Isolation has a cost. vLLM says input preparation, sampling and KV cache management are excluded, because they are shared across most deployments and changing them for one workload risks breaking other stable configurations. Work there should involve core maintainers more closely.

Bounded scope does not mean dropping features. For K3, initial hardware validation covers GB300 with TP8 and TP16, decode batches up to 64, speculative decoding with various draft lengths, and decode context-parallel configurations within one NVLink domain. Full-context serving, temperature sampling and structured output are treated as first-class features that belong in the validation plan.

What it means for vLLM users and contributors

For operators running Kimi K3 on GB300 hardware within the supported configurations, the reported gains point to higher decode throughput and better interactivity. The benefits apply only where the optimization is defined to apply.

For contributors, vLLM says agents should build tests, run evaluations and benchmarks, investigate failures and document the conditions under which an optimization works. Human reviewers would still judge the design and the evidence, but from a higher level than a line-by-line code review. The team says it believes the community can build this together, bringing more models, GPUs and workloads closer to the speed of light.