Push-based KV transfer in vLLM cuts TTFT up to 3x in AWS benchmark

Why KV transfer matters in disaggregated inference

vLLM is an open-source inference and serving engine for large language models. LLM inference has two phases: prefill, which processes the input prompt and builds the key-value (KV) cache, and decode, which uses that cache to generate output one token at a time. Because the phases differ in compute, memory and scaling behavior, prefill-decode (P-D) disaggregation runs them on separate engine instances so each can scale and be optimized on its own. According to AWS, this can improve resource utilization, throughput and latency under load.

The catch is that the KV cache must move from the prefill instance to the decode instance before token generation can start. That makes the efficiency of the transfer central to TTFT.

vLLM can do this through the NVIDIA Inference Xfer Library (NIXL) connector, which moves KV blocks over RDMA-capable interconnects such as AWS Elastic Fabric Adapter (EFA). On supported instance types, EFA supports GPUDirect RDMA and Neuron Direct RDMA, so NIXL can transfer blocks directly between GPU or AWS Trainium device memory without staging through host memory.

How pull mode falls short and what push changes

The NIXL connector defaults to pull mode. After prefill finishes, the router relays source metadata, including prefill block IDs, to the decode instance. The decode instance then allocates destination blocks, prepares the transfer and issues a NIXL READ. These steps run one after another after prefill and before decoding, so they lengthen the path to the first token.

The push-based connector changes the order of work. The decode instance registers its target blocks in advance, while the prefill instance is still computing the KV cache. Transfer starts as soon as the producer's blocks are ready, which removes the consumer-side ID-discovery round trip. Transfer setup also moves to the prefill side and runs on a background thread, so the decode instance does not spend cycles on it during token generation. AWS says the net effect is lower TTFT and higher output-token throughput on the decode instance.

The protocol has three steps:

• Registration (decode side): when blocks are allocated for a request needing remote prefill, the decode worker sends the prefill instance a NIXL notification describing where to write the KV.
• Rendezvous (prefill side): registration and prefill completion can arrive in either order. Whichever comes second triggers the transfer, and requests are paired by ID, including preempted or retried ones.
• Transfer (prefill side): the producer connects to the decode instance if needed and issues a NIXL WRITE directly into the decoder's registered memory. The decoder treats the KV as received once it has counted one completion notification per producer rank.

Producer and consumer can use different internal block layouts, so block sizes may differ on each side.

Threading model and routing tradeoff

Each prefill worker uses a dedicated background transfer thread. When both the destination registration and the prompt KV are ready, the prefill main loop hands the blocks to that thread and returns to model execution. The thread establishes or reuses the NIXL connection, maps block IDs to memory descriptors, merges contiguous blocks into larger transfer regions and submits the WRITE. Keeping this per-request work out of the latency-critical loops prevents it from delaying model execution.

Both modes use the same NIXL and EFA/RDMA data path. The difference is that pull performs metadata relay and decode-side setup after prefill, while push overlaps decode-side setup with prefill.

There is a routing tradeoff. Pull lets the router pick a decode instance after prefill completes, which gives more flexibility for load-aware routing and quality-of-service management. Push chooses the decode instance earlier so setup can overlap with prefill. In return, under comparable load, the prefill instance can release its KV blocks sooner.

Benchmark results and what they mean for vLLM users

AWS evaluated the feature with Qwen3-32B on a 1P1D deployment using Amazon EC2 P5en instances connected with EFA. Push mode delivered up to 3x better TTFT and up to 30 percent better TPOT than pull mode, and AWS notes that results depend on workload and configuration. It adds that push mode helps particularly at high request rates.

AWS built and validated the feature on AWS Trainium clusters connected with EFA, then contributed it to vLLM as an accelerator-agnostic feature. The P5en GPU results are meant to show that the gains carry over to GPUs.

For teams running disaggregated prefill/decode inference with vLLM, push mode is an option to improve responsiveness, especially under heavy load. Those who rely on load-aware routing after prefill should weigh that against the earlier decode instance selection push requires.