vLLM Expands Encoder Disaggregation for Multimodal AI Serving
vLLM has detailed engineering work that separates multimodal encoding from language-model prefill and decode, allowing operators to scale those stages independently. The project, known as Encoder, Prefill, Decode, or EPD, adds metadata-only requests, remote embedding transfers and a dedicated encoder execution path.
- EPD separates encoder, prefill and decode deployments.
- Metadata-only requests avoid repeated multimodal media processing.
- An audio evaluation reached 73.13 requests/s with metadata-only forwarding.
- A Kimi-K3 encoder loaded 2 of 96 checkpoint shards.
- Core capabilities are included in vLLM v0.32.0.
Separating multimodal processing from language generation
Multimodal requests typically pass through three stages. The encoder preprocesses images, video frames or audio and produces embeddings. Prefill processes the prompt and those embeddings, while decode generates output tokens.
vLLM supports two EPD deployment patterns: E+PD, where encoding is separate from a combined prefill and decode instance, and E/P/D, where all three stages are deployed separately. Encoder connectors transfer embeddings, while KV connectors transfer the state between prefill and decode.
This separation lets operators adjust instance counts and execution settings for each stage. An image-heavy workload can receive more encoder capacity without a corresponding increase in decode capacity, while workloads producing longer responses can be configured differently.
Metadata-only requests reduce repeated media processing
Moving embeddings between instances does not by itself provide all the information needed to reconstruct a multimodal input. The downstream instance may still need placeholder lengths, image grids, feature lengths and identifiers that describe where the embeddings belong.
vLLM's metadata-only request path replaces the downstream media item with a reference containing structural information produced by the encoder. The consumer reconstructs the input layout from that metadata and receives the embeddings through an encoder connector. The text prompt and generation settings remain unchanged.
The Python EPD path supports this handoff for images, video and audio in supported models. For example, Qwen-family image inputs can use image_grid_thw, while Qwen3-VL video inputs can use video_grid_thw and encoder-published timestamps. Audio paths can transfer token counts and feature lengths needed to rebuild placeholders.
In an evaluation using Qwen3-Omni-30B-A3B-Instruct, metadata-only forwarding increased throughput from 62.21 to 73.13 requests/s and reduced mean latency from 980 to 853 ms. Accuracy changed from 75.90% to 75.65%, a difference vLLM says remains under investigation.
Dedicated encoder execution and broader serving integrations
Model Runner V2 now provides a dedicated MMEncoderModelRunner for encoder-only instances. These instances do not run the language model, maintain a language-model KV cache or produce sampled tokens, so the dedicated path manages GPU completion without relying on token sampling.
Related changes include automatic encoder-only mode for producer-only encoder connector instances, reduced startup I/O and opt-in encoder CUDA Graph capture. On a pruned Kimi-K3 checkpoint, the encoder read 2 of 96 shards and weight loading dropped from 683 s to 0.31 s.
vLLM also describes remote embedding transfer through NIXL and Mooncake, cross-encoder output reuse through Mooncake Store, and lower orchestration overhead in the Python EPD proxy. EPD orchestration is being integrated with vLLM Router, SMG and llm-d.
The core capabilities described, including MiMo and XPU support, are included in vLLM v0.32.0. DeepSeek V4.1 Flash metadata-only support is available on main through #62802, while the vLLM Router and SMG integrations use their respective main branches. For users and operators, the changes provide more ways to distribute multimodal serving work according to the demands of media processing, prompt handling and token generation.
