vLLM Guide Shows When Splitting Prefill and Decode Pays Off
The vLLM project has published a practical guide to disaggregated serving, which separates prompt processing, token generation and CPU-side work into different tiers. According to the guide, the split stops long prompts from stalling other users' output, as long as the KV cache moves between tiers fast.
- vLLM's guide covers splitting prefill, decode and CPU-side rendering into separate tiers, with examples requiring vLLM v0.30.0 or later.
- On two L40S GPUs, collocated p99 ITL reached 169 ms at 0.4 req/s, while prefill/decode stayed between 25 and 52 ms.
- AMD's MoRI-IO benchmark showed 73 of 100 requests meeting targets versus 30 of 100 collocated, about 2.4 times the goodput.
- Without fast KV transfer, the test setup's median TTFT rose to 2.2 s versus 0.7 s collocated at 0.2 req/s.
- The guide advises staying collocated when TTFT is the main constraint or traffic is low and bursty.
Three jobs in one process
A plain vllm serve process handles three workloads with little in common, the guide says. Prefill reads the whole prompt in one pass, is limited by compute and sets time to first token (TTFT). Decode produces one token at a time and is limited by how fast the GPU reads model weights from memory, which sets inter-token latency (ITL). When both share a GPU, every decode stream waits while a long prefill runs.
The third job is easier to miss: chat templating, tokenization, detokenization, reasoning parsing and tool call parsing. This is pure CPU work running on a machine rented for its accelerators.
Where the work can be split
vLLM offers several split points that can be combined. Prefill and decode can run as two instances, with the KV cache passed between them. The guide gives a sense of its size: Llama-3.1-70B in BF16 stores 320 KiB per token, so a 10k-token prompt hands decode about 3 GB. That is roughly 65 ms at a 400 Gb/s line rate before any overhead, and all of it lands on TTFT.
A KV connector does the moving, usually over RDMA. The guide says there are more than a dozen upstream, including NIXL, LMCache, Mooncake, FlexKV and AMD's MoRI-IO, plus a MultiConnector that chains them.
The frontend can also leave the GPU box. The /render endpoint turns an OpenAI request into token IDs, the engine works purely on tokens, and /derender turns output token IDs back into a response with content, reasoning and tool calls separated. Beyond prefill/decode, the guide mentions encoder disaggregation for multimodal models and an AFD plugin that splits attention from FFN in MoE models. Prefill/decode now also covers hybrid SSM models, including Mamba state transfer.
What the benchmarks show
The guide stresses that the benefit is goodput, meaning the request rate sustained while requests still meet both TTFT and ITL targets, not peak throughput.
In the authors' test on one box with two NVIDIA L40S GPUs, running Qwen2.5-7B-Instruct with roughly 8k-token prompts, collocated serving saw p99 ITL jump from 23 ms to 169 ms at 0.4 req/s and reach 263 ms at 2 req/s. The prefill/decode setup kept p99 between 25 and 52 ms. Median ITL was 21 to 24 ms in both setups up to 1 req/s, and at 0.4 to 0.6 req/s the collocated p99 was about six times higher. The guide notes that its collocated baseline is not the strongest possible one.
Other results come from elsewhere. In AMD's single-node MoRI-IO benchmark with Qwen3-235B-A22B-FP8 at 8 req/s on one 8-GPU MI300X node, 73 of 100 requests met both a 1 s TTFT and a 50 ms ITL target, against 30 of 100 for collocated serving, about 2.4 times the goodput. The guide also cites llm-d's report of about 59% lower mean end-to-end latency and 67% lower P95 for gpt-oss-120b on 16 H200s.
The cost: every first token pays for the transfer
Those gains assume fast KV movement, such as RDMA over InfiniBand or RoCE between nodes, or NVLink or GPU peer-to-peer within one. The authors' test box had neither. Each 8k prompt carried about 470 MB of KV cache and took about 1.3 s to reach decode. At 0.2 req/s the median TTFT was 2.2 s for prefill/decode against 0.7 s collocated, so it missed a 2 s TTFT target at every rate even though its ITL tail stayed flat. The guide attributes most of the delay to overhead around the copy rather than bandwidth.
Its advice is to check the transfer first. On one machine, nvidia-smi topo -p2p r should report OK between the prefill and decode GPUs. Then send a few long prompts one at a time and read the KV Transfer metrics line on the decoder. If the average transfer time is in the hundreds of milliseconds, fix that before benchmarking anything else.
The CPU tier is cheaper. Templating and tokenizing a 9k-token chat prompt for Qwen2.5-7B cost about 15 ms of CPU, and one render server with default settings topped out at 73 req/s using just over one core. At 0.4 req/s that is under 1% of one core.
Who should use it
The guide recommends disaggregating when ITL p99 misses your service target under production load, and for long prompts at high concurrency if KV transfer is fast. Chat or agent loops over a growing context are also candidates, paired with KV offloading or a shared cache such as LMCache or Mooncake. Teams that see templating, tokenizing or parsing in their GPU nodes' CPU profile can split off only the render tier.
It advises staying collocated, or measuring first, when TTFT is the binding constraint, when KV transfer is slow, or when traffic is low, bursty or latency-insensitive. The trade-off is operational: you run three or four services instead of one, and KV transfer becomes a new failure mode.
The hands-on part of the guide assumes vLLM v0.30.0 or later and uses Qwen3-0.6B for wiring checks, though it says that model is too small to show a real benefit. The basic setup is a prefiller, a decoder and a proxy, with the prefiller configured with the NixlConnector in the kv_producer role.
