Baseten cuts GLM-5.3 delta weight sync from 36 to 15 seconds

From a CPU copy to the GPU as the parent

In delta weight syncs, a trainer in the customer's environment publishes full checkpoints and sparse deltas to object storage. According to Baseten, each delta tensor is zstd(parent_bytes XOR new_bytes). Once a snapshot is uploaded, the trainer sends a sync request naming the policy it wants served. Rollout clusters pull each object once through the Baseten Delivery Network (BDN), reconstruct the new policy and update their GPUs in place. Trainer and rollout GPUs share no infrastructure, which is unchanged.

The first design kept the complete parent checkpoint in host memory, because the decoder needed it to turn XOR bytes into new values. Baseten lists three costs. Each replica held about 770 GB of host RAM for GLM-5.3. On an 8×B300 node, that copy and a 1.1 TB host KV cache could not coexist, so the cache had to shrink to 275 GB. Promoting the CPU copy to the new policy took about 45 seconds per update and blocked the next delta. And every update had to land on both the GPU and the CPU copy, creating two sources of truth.

In the new design, the model is loaded from a local disk copy at startup. Every served tensor is verified against it through the engine's GPU layout, a per-tensor Adler-32 checksum is recorded, and the copy is deleted. From then on, GPU memory holds the only parent.

How the update works

When a delta arrives, Baseten's native decoder walks zstd's commands directly and keeps only the nonzero XOR bytes: the position and XOR mask of each changed element. It never builds the dense XOR tensor and never reads model weights. While the old policy keeps serving, each GPU worker gathers old values at the changed positions, applies the XOR mask to get absolute new values, verifies the result against the trainer's checksum and seals the values into a write plan in the engine's native tensor layout.

The checksum step is also cheaper. Because each byte contributes to Adler-32 independently, depending only on its position and value, a changed byte's effect can be computed in constant time. Baseten starts from the checksum recorded at startup, adds one correction per changed byte and compares the result with the trainer's. According to the company, this made the whole update up to 2.3× faster, with a Kimi K3 update on a 30 GB delta going from about 230 to about 100 seconds. A corrupted download or wrong parent is rejected before any GPU write.

Inside the pause, the system closes admission and pauses running requests in place, writes new values and refreshes derived weights, then publishes the new policy version and reopens admission. On GLM-5.3 the pause is 0.8 seconds, down from 6, and the 45-second CPU promotion is gone.

Because update work now shares GPUs and CPUs with serving, Baseten added further measures:

• A parallelism-aware decoder that splits each delta by the GPU owning each change, mapping changes to addresses in that GPU's packed layout, including swizzled FP8 and FP4 scale factors.
• Update planning in a separate, CUDA-free helper process, so CPU-heavy work does not contend for the GIL in SGLang's Python scheduler loop.
• Pipelined download and decode, with each shard decoded as it lands, on low-priority native threads sized to leave 16 cores for serving.

Kimi K3 on GB300 and KV cache offloading

Kimi K3 is 1.7 TB in its served MXFP4 layout. With a CPU checkpoint, each GB300 node would need roughly 838 GB of host memory for its half of the weights, nearly a serving container's entire memory limit, according to Baseten. The company says this was a hard constraint because GB300s come with pre-configured hardware. The new design runs on two GB300 nodes with four GPUs each. Each node decodes, verifies and stages only the tensors it owns, and both must write successfully before the new policy is published. On an 8.4 GB delta, Kimi K3 goes from sync request to live weights in 75 seconds, with a 4.1-second pause.

The freed host memory can also serve as a KV cache tier. Evicted GPU KV pages are written back to host memory instead of dropped, so a later request with the same prefix reloads them rather than re-running prefill. Baseten says reloading an evicted 150k-token prefix from CPU memory takes 0.48 seconds, versus 4.9 seconds of fresh prefill.

On the AgentX benchmark with Kimi K3, Baseten compared offloading on the new design against the old CPU-checkpoint setup. At a concurrency of 16 the KV cache fits on the GPU, so there is no difference. At a concurrency of 48, the hit rate rises from 2% to 92% and output throughput per GPU rises 3.2×.

What it means for RL rollout users

For teams running reinforcement learning rollouts, shorter syncs and pauses mean the served policy catches up with the trainer faster, and the next delta can be applied as soon as it lands. Baseten says the design also unlocks Kimi K3 RL rollout on 8:GB300, which was previously blocked by CPU memory.

Baseten notes that all runs used one replica with trainer-shaped fixture deltas, timed from the trainer's sync request to the first new response. The company points interested teams to its RL rollouts docs or its team.