Amazon EC2 P5en
vLLM adds push-based KV transfer to cut TTFT in disaggregated inference
vLLM has introduced a push-based mode for its NIXL connector that moves post-prefill transfer coordination off the critical path in disaggregated inference. In a…
vLLM adds bi-directional KV transfer to cut TTFT in multi-turn agent workloads
vLLM has added bi-directional KV cache transfer for prefill-decode disaggregated serving, letting decode nodes send previously computed context back to prefill nodes on…