vLLM cuts Elastic EP scaling pause from 100 seconds to 0.82 seconds

Why scaling used to be costly

According to vLLM, changing the number of GPUs in a service traditionally meant adding or removing whole model replicas, or restarting the engine with a different GPU count. As models grow, each replica needs more GPUs, so scaling by replica happens in ever larger steps.

The project gives a concrete example: restarting vLLM to scale DeepSeek V4 Pro from 16 to 20 GB200 GPUs stopped serving for 197 seconds, leaving expensive GPUs allocated without generating tokens.

The first Elastic EP release let vLLM add or remove GPUs in a running DEP deployment (data-parallel attention with expert-parallel MoE) without a restart. Serving still paused, though, while new workers were initialized, communication groups prepared, weights loaded, and kernel warmup and CUDA graph capture repeated. In the same benchmark that pause lasted 100 seconds when scaling from 16 to 20 GPUs and 56 seconds when scaling back to 16.

Two phases: background preparation and a short commit

vLLM reorganized reconfiguration into background preparation and a brief commit. Existing workers use separate background threads to prepare and warm up the target communication, transfer model weights, and synchronize execution information such as autotuning results with joining workers. Meanwhile they keep serving requests with their current communication groups and CUDA graphs.

Joining workers, in parallel, initialize, receive weights and prepare their communication and GPU execution. During the commit, vLLM briefly pauses its schedulers, activates the prepared topology and resumes serving. Preparation covers four tasks:

• Initialize joining workers: start processes and allocate storage for the model and execution.
• Prepare communication: set up target communication groups and expand the EP communicator.
• Transfer model weights: move weights from existing workers over NVLink or RDMA.
• Prepare execution: finish compilation, kernel warmup, tuning setup and CUDA graph construction on joining workers.

After the commit, asynchronous Expert Parallel Load Balancing (EPLB) lets new workers begin handling attention while existing workers handle expert computation. EPLB then redistributes experts across the expanded group in the background.

NIXL EP and preserved CUDA graphs

Asynchronous preparation alone was not enough. Earlier EP communication backends in vLLM used fixed membership, so adding or removing ranks required a new instance with its own buffers, memory registrations, IPC handles and RDMA Queue Pairs. Switching communicators also invalidated captured CUDA graphs. According to vLLM, even with asynchronous preparation, recapturing graphs during the commit caused pauses of 58 seconds on scale-up and 44 seconds on scale-down.

To address this, vLLM designed NIXL EP, an EP communication library built for Elastic EP. It adds or removes ranks in place, reusing the same NIXL agent, buffers and connections, and only joining peers need new connections and registrations. It also keeps captured EP dispatch and combine entries valid, so retained workers can reuse their CUDA graphs.

vLLM also changed its own code so the surrounding compute kernels keep stable arguments, launch configurations and tensor addresses. It reuses fused-MoE execution objects on retained workers, keeps addresses and layouts of graph-visible workspaces and dispatch outputs fixed, and preserves the storage of expert-routing tensors while updating mappings in place.

Joining workers prepare independently. NIXL EP lets each one mask every remote EP peer during warmup and graph capture, and vLLM no longer makes these passes wait for DP batch information from other workers. After capture, peers can be reactivated without invalidating the graphs.

Results and what they mean

On DeepSeek V4 Pro with GB200 GPUs, TP=1 and DP/EP sizes of 16 and 20, vLLM reports downtime of 0.82 seconds when scaling from 16 to 20 GPUs and 1.96 seconds when scaling back to 16. The project says this allows scaling up and down with almost no impact on serving performance.

For operators of large MoE deployments, the practical point is that capacity can be adjusted in smaller GPU increments without a long stop in token generation. These figures come from vLLM's own benchmark on a single model and hardware setup, and the source text provided does not include further measurements.