vLLM adds push-based KV transfer to cut TTFT in disaggregated inference

Why KV transfer matters in disaggregated inference

vLLM is an open-source inference and serving engine for large language models. LLM inference has two phases: prefill, which processes the input prompt and builds the key-value (KV) cache, and decode, which uses that cache to generate output one token at a time. Because the phases differ in compute, memory and scaling characteristics, prefill-decode (P-D) disaggregation runs them on separate engine instances so each can scale and be optimized independently.

The catch is that the KV cache must move from the prefill instance to the decode instance before token generation can start. That makes efficient transfer critical to TTFT. One way vLLM does this is the NVIDIA Inference Xfer Library (NIXL) connector, which moves KV blocks over RDMA-capable interconnects such as AWS Elastic Fabric Adapter (EFA). On supported instance types, EFA supports GPUDirect RDMA and Neuron Direct RDMA, so NIXL can move KV blocks directly between GPU or Trainium device memory without staging through host memory.

The connector's default mode is pull-based. After prefill finishes, the router relays source metadata, including prefill block IDs, to the decode instance. The decode instance then allocates destination blocks, prepares the NIXL transfer and issues a NIXL READ. Since these steps run one after another before decoding can begin, they lengthen the TTFT critical path.

How the push-based connector works

In push mode, the decode instance registers its target blocks in advance, while the prefill instance is still computing the KV cache. Transfer starts as soon as the producer's blocks are ready, which removes the consumer-side identifier-discovery round trip from the path to first token. Descriptor preparation and transfer submission also run on the prefill side, on a background thread, so the decode instance does not spend those cycles during token generation.

The protocol has three steps:

• Registration (decode side): when the decode instance allocates local blocks for a request needing remote prefill, it sends a registration with its identity and the reserved block locations to the prefill instance as a NIXL notification.
• Rendezvous (prefill side): registration and prefill completion can arrive in either order. Whichever happens second triggers the transfer. Requests are paired by identifier, and the matching accounts for preempted or retried requests.
• Transfer (prefill side): the producer connects to the decode instance if needed and issues a NIXL WRITE that copies KV blocks directly into the decoder's registered memory, addressing each destination tensor-parallel rank.

Producer and consumer may use different internal block layouts, since each side maps block locations to its own physical layout at transfer time. Each WRITE carries a completion notification. The decoder counts one per producer rank and treats the KV as received only when all expected notifications have arrived.

A dedicated background transfer thread on each prefill worker keeps this work away from model execution. It establishes or reuses the NIXL connection, maps KV block IDs to memory descriptors, merges contiguous blocks into larger transfer regions and submits the WRITE. The prefill main loop hands off the blocks and quickly returns to model execution.

Benchmark results and routing tradeoff

vLLM evaluated the feature with the Qwen3-32B model on a 1P1D deployment using Amazon EC2 P5en instances connected with EFA. To isolate prefill-to-decode transfer performance, prefix caching was disabled with --no-enable-prefix-caching, so the decode node retrieves the full KV cache from the prefill node for every request. In this setup, push mode delivered up to 3x better TTFT and up to 30% better TPOT than pull mode, particularly at high request rates. vLLM notes that results depend on workload and configuration.

The feature was built and validated on AWS Trainium clusters connected with EFA, then contributed to vLLM as an accelerator-agnostic feature. The P5en results show the gains carry over to GPUs, according to the post.

There is a tradeoff. Pull mode lets the router choose a decode instance after prefill completes, which gives more flexibility for load-aware routing and quality-of-service management. Push mode picks the decode instance earlier so destination setup can overlap with prefill. In return, under comparable load, the prefill instance can release its KV blocks sooner because the transfer begins once both the prompt KV and destination registration are ready.

What it means for users

For teams running disaggregated prefill/decode inference with vLLM, push mode can improve application responsiveness by reducing TTFT and TPOT, according to vLLM. The gains are most relevant at high request rates, and the routing tradeoff means the choice between pull and push depends on how much a deployment values flexible late routing versus lower latency.