vLLM Makes RL Training and Inference Bitwise-Identical on DeepSeek-V4-Flash

The problem: optimized rollouts drift from training

Rollout dominates RL training time, so training systems adopt low-precision compute, a low-precision KV cache, and communication and kernel optimizations. According to vLLM, these changes alter the numerics on the inference side. The inference engine generates a response, and the training engine recomputes its probabilities and updates the model. Even with identical weights, the two sides can assign different probabilities to the same token. Training then optimizes a policy other than the one that produced the samples.

vLLM says this gap can grow with updates and destabilize training. In its DeepSeek-V4-Flash-Base experiment, run without algorithmic corrections such as TIS, the high-performance baseline's train-inference KL rose from 2e-3 to 29.9 while reward fell back. Unstable runs mean more crashes, rollbacks and reruns, and more total time to reach the same result.

The approach: shared kernels and an exact-equality bar

In vLLM's method, the training forward pass calls vLLM's inference kernels directly. Those high-performance kernels are modified so results do not change with batch composition, prefill or decode phase, or parallel layout. The team defines "diff" as the difference between the rollout logprob and the training side's recomputed logprob for each valid sampled token, measured before each update with the same weight version and token prefix. Acceptance requires the max, mean and standard deviation of the per-token difference, and the k3 KL, to be exactly 0. According to the post, this is exact equality, not a threshold.

The post compares the approach with existing options. Algorithmic corrections such as TIS, MIS, FlashREINFORCE and Score Centering leave the inference engine alone and are cheap to adopt, but the numerical difference remains. R3 replays rollout expert routing and removes only routing flips. QAT lets rollout use low precision but requires training to simulate quantization. Generic batch-invariant kernels achieve strict alignment but cost speed: in the Thinking Machines workload cited by vLLM, time went from 26 s to 55 s, or 42 s with an improved attention kernel. vLLM says aligning on the high-performance path keeps speed but costs per-model, per-operator effort. It set a bar of at least 70% of an equally optimized baseline's rollout throughput.

Results on DeepSeek-V4-Flash and Nemotron

Over 100 DeepSeek-V4-Flash training steps, vLLM reports diff of 0 at every step at 91.3% of the baseline's rollout throughput. The team then carried the method to Nemotron-3.5, a Mamba-2 hybrid using NVFP4. There, the consistent run's SWE-bench Pro score holds, while the baseline's drops from 0.56 to 0.22.

vLLM stresses that this post measures numerical consistency, training curves and rollout throughput. The expected end-to-end gain comes from fewer crashes and reruns, which only shows over long runs, and that wall-clock comparison is left for later.

An agent skill to lower the cost of new models

The weak point of strict alignment is cost on each new model. To address it, vLLM turned the process into an agent skill. It instructs an agent how to find the first operator where training and inference diverge, reduce it to a minimal repro, fix it and rerun the regressions. The skill is released with the recipes and code.

Work is continuing. According to vLLM, DeepSeek-V4.1 has reached 0 diff, and performance work is being finalized and will get its own post.

What it means

For teams running RL on large models, the claim is that exact train-inference agreement can coexist with fast, low-precision rollouts, removing a source of unstable runs. vLLM also notes that a bitwise-consistent run can serve as a baseline for measuring other corrections, such as those used for stale weights in asynchronous training. Whether the saved reruns translate into shorter total training time is not yet shown, since vLLM leaves that comparison for later.