vLLM says aligned kernels make asynchronous RL stable at lag 64

Asynchronous training has two different gaps

Asynchronous reinforcement learning allows rollout and training to run at the same time. This can improve throughput, but the learner updates samples generated by older model weights. In the REINFORCE Zero experiments, the policy lag reached 2 and 8 on math reasoning tasks, and 64 on agentic tool use.

vLLM separates the resulting logprob difference into weight lag and kernel mismatch. Weight lag compares different policy versions and changes smoothly across recent updates. Kernel mismatch occurs when the rollout engine and learner produce different values for the same weights and tokens. According to vLLM, this second gap fluctuates with batch composition and numerical execution, and importance sampling alone does not remove it.

In the cited experiments, aligned kernels produced an importance-weight deviation peak of 1.50 times the median, compared with 442.54 times for standard kernels. The figures come from the REINFORCE Zero paper, by NVIDIA Research and Inferact.

REINFORCE Zero removes the mismatch first

The recipe has three parts: align the rollout and learner kernels, apply token-level importance sampling without clipping, and optimize plain REINFORCE. The rollout records each sampled token's logprob, while the learner divides its own probability by that recorded value.

Each response uses an advantage equal to its reward minus the mean reward of its group. Tokens are averaged within a response. The method uses no value model and no rejection gates.

vLLM says clipping the importance weights can introduce bias. Once the kernel term is removed, the company argues that unclipped importance sampling can correct the remaining weight lag while avoiding the large fluctuations seen with standard kernels.

Reported results include 2.40 times higher throughput

With aligned kernels, REINFORCE Zero remained stable through 20,000 updates in the cited 1.5B reasoning experiment. At policy lag 64, a Qwen3-4B setup with Python tools and a 16K context continued learning through 5,000 updates. Mean reward rose from 0.287 in the first 100 updates to about 0.78.

Across 16 GPUs and 12,000 updates, vLLM reports 2.40 times the training-step throughput of synchronous training with aligned kernels, with the listed runtimes at 68.55 and 164.77 hours. In another experiment, a Qwen3-30B-A3B setup with tools improved macro pass@1 from 31.29% to 71.55% in 3,000 updates.

By contrast, asynchronous importance sampling with standard kernels saw rollout-to-learner KL rise 83.33 times between the first and last 1,000 updates. Online AIME24 pass@1 fell from 34.27% to 13.96%, while the largest gradient norm before clipping reached 38.27, compared with 0.0555 for aligned kernels.

Zero mismatch expands to newer models

vLLM says its zero-mismatch work now covers DeepSeek-V4.1-Flash, Kimi K3.1 on day 0, and both DeepSeek-V4-Flash and Kimi K3.1 on AMD MI355X. For DeepSeek-V4.1-Flash, the consistent path uses FP8 and MXFP4 weights, MegaMoE, sparse attention with a sliding window, DSpark speculative decoding and CUDA graphs. vLLM says the DAPO rollout step was within 10% of the fast baseline.

The company says the alignment code and related materials are available in Inferact/zero-mismatch-rl. The results suggest that asynchronous RL users can pursue higher training throughput without adding sample-rejection gates, provided their rollout and learner computations are aligned. vLLM also reports a cost: on OSWorld, aligned kernels had a mean step time of 944.25 seconds, compared with 884.17 seconds for standard-kernel importance sampling with a sequence gate.