vLLM pairs PCP and DCP to serve 1M-token GLM-5.3 contexts on 16 H200s

The problem: long agentic contexts on H200s

In the second part of its GLM-5.3 optimization series, vLLM describes two limits that long agentic workloads hit on H200 GPUs: KV cache memory and prefill speed. Agentic contexts grow with every turn, as tool output and file contents are added. On H200s, GLM-5.3's weights take up most of each GPU's memory, leaving little room for the KV cache. Under load, the prefill engines can also become the bottleneck for the whole deployment.

The setup uses 16 H200s split into one prefill node and one decode node. Part 1 of the series showed how Hybrid HiSparse lets long contexts fit in an aggregated deployment on a single 8x H200 node. Part 2 extends the approach to prefill/decode disaggregation.

PCP and DCP on the prefill node

vLLM says the usual options fall short. With DP8, each prompt is handled by one GPU, which limits time to first token for long prompts and forces the whole KV cache into that GPU's memory. With TP8, attention heads are split across GPUs, but MLA's latent KV is shared by all 64 heads, so every GPU stores the full cache. The indexer, which selects the 2,048 tokens that sparse MLA attends to, is not partitioned, so its quadratic cost is repeated on every GPU.

PCP instead splits each prompt by token. Every GPU computes the queries for one eighth of the prompt, with all 64 heads. To balance the causal attention load, the prompt is cut into 16 chunks and each GPU takes one from the front and one from the back. According to vLLM, this raises sparse MLA from about 26 to about 191 FLOPs per byte moved. It also moves the point where the indexer overtakes sparse MLA in a from-scratch prefill from about 32K tokens under TP8 to about 256K under PCP8.

PCP still leaves every GPU storing the full KV cache. A 1M-token GLM-5.3 request needs 53.4 GiB of FP8 KV, but each GPU has room for only about 15 GiB. DCP deals tokens round-robin across the eight GPUs, so the same request needs 6.7 GiB per GPU. The cost is an all-gather of the cached context every layer. vLLM measured DCP8 adding about 6% per step up to 64K tokens of context, rising to 15% at 248K. It says it is working on overlapping the all-gather with the previous layer's compute. Beyond roughly 250K tokens, PCP8 alone cannot hold the KV cache, so DCP is what makes long prompts possible.

Deployment and benchmark

The deployment runs on two 8x H200 nodes, with an FP8 KV cache and MTP speculative decoding with three draft tokens on both engines.

• Prefill: one PCP8 x DCP8 x EP8 engine, with 512 GiB of CPU KV offload so later turns reuse their prefix.
• Decode: eight DP ranks with EP8 and Hybrid HiSparse (48 GiB host pool plus 16 GiB offload per rank), so concurrency is bounded by host memory rather than HBM.

A prefix-aware llm-d router sends each request to the prefill engine, the NIXL connector moves the KV cache to a decode rank, and that rank streams tokens back. vLLM compared this against DP8 prefill and against plain CPU KV offloading on the decode node, using the same 16 GPUs, host memory and vLLM build.

The test used the SemiAnalysis AgentX benchmark, which replays Claude Code traces with prefix reuse, with one-hour runs on a fresh server per concurrency level. The MTP acceptance length was fixed synthetically at 2.99 so all configurations get the same speculative speedup.

What it means

According to vLLM, PCP8 x DCP8 prefill with Hybrid HiSparse decode gives 1.7 to 3.2 times the per-user speed of a data-parallel deployment at the same throughput. It is also the only configuration the team tested that serves 1M-token contexts. For operators running long-context agentic workloads on memory-constrained H200 hardware, the results point to token-level parallelism and host-memory KV storage as ways to gain both speed and context length. The figures come from vLLM's own benchmark setup.