vLLM adds bi-directional KV transfer to cut TTFT in multi-turn agent workloads

The problem with one-way KV transfer

Multi-turn agentic workloads build up context as an agent reasons, calls tools and uses earlier results. On each turn the client resends the whole conversation history, including system instructions, prior prompts, tool results and model responses, so the growing context is processed again and again.

In prefill-decode (P-D) disaggregation, the prefill node processes the prompt and builds its KV cache, and the decode node then retrieves those KV blocks to generate the response. Conventional setups transfer KV cache only in that one direction, from prefill to decode. According to vLLM, this causes two inefficiencies:

• After generating a response, the decode node holds KV entries for both the prompt and the new tokens. The prefill node cannot retrieve them on the next turn, so it must recompute the KV projections for the previous response tokens.
• The decode node maintains session state while prefill nodes do not, which raises the chance that active sessions are evicted from the prefill cache. On a full cache miss, the prefill node recomputes the entire conversation history.

As conversations grow, this redundant work uses more accelerator capacity and increases TTFT.

How bi-directional transfer works

The mechanism lets a decode node return previously computed KV to a prefill node on later turns, so shared context is not recomputed. vLLM extended its NVIDIA Inference Xfer Library (NIXL) KV connector so that prefill nodes can consume remote KV blocks, much as decode nodes already do. The team says this was not trivial, because the two node types differ: prefill nodes can provide complete prompt KV blocks, while decode nodes can only serve previously cached blocks. The transfer itself, a NIXL READ, is common to both the P-to-D and D-to-P directions.

The flow has six steps added to the existing path:

• The decode node retains eligible KV blocks after generation.
• It returns their metadata in the streaming response.
• The router caches that metadata for the next prefill request.
• The prefill node checks the remotely reusable KV after applying local cache matches.
• It queues an asynchronous receive when the reusable token count meets the threshold.
• It performs a D-to-P NIXL READ before computing the remaining suffix.

Coordination relies on a KV-cache-aware router, a lightweight, stateful control-plane component. It maps the client-provided conversation_id to decode's kv_transfer_params. The router stores only block-location metadata, and KV tensors move peer-to-peer rather than through it. vLLM's GitHub provides an example proxy that implements this flow.

Retention and recompute controls

Normally a decode instance frees a request's blocks once generation finishes. In bi-directional mode it holds them for a configurable lifetime, decoder_kv_blocks_ttl, which defaults to 480 seconds. A turn arriving before expiry reuses the KV; a later one falls back to normal prefill. Because the TTL is fixed and not renewed, vLLM advises tuning it to the expected time between turns to balance reuse against accelerator-memory use. The team names extending a heartbeat-renewed lease to the decode side as a natural next improvement.

A second setting, kv_recompute_threshold, defaults to 64 tokens. For short reusable prefixes, interconnect and coordination overhead can exceed the cost of recomputing, so the prefill node recomputes when the token count to compute is below the threshold and pulls KV from decode when it is at or above it.

Results and what it means

The benchmarks used a 1P1D deployment on Amazon EC2 P5en instances connected with AWS Elastic Fabric Adapter (EFA). vLLM says results depend on workload and configuration. The team originally built and validated the feature on AWS Trainium clusters with EFA and contributed it to vLLM as an accelerator-agnostic capability; the GPU results show the benefits carry over.

For teams building conversational assistants and agentic applications on disaggregated serving, the change targets latency and accelerator capacity lost to repeated prefill as histories lengthen. Operators would need a KV-cache-aware router and should tune the TTL and recompute threshold to their traffic.