Artificial intelligence news.
Straight from the original announcements.

Google details bitwise-aligned TPU training for stable LLM reinforcement learning

November 16, 2026By the AI News Bites Editorial TeamSource: Google for Developers
Google details bitwise-aligned TPU training for stable LLM reinforcement learning
Image: Google for Developers

Google has described an open-source Tunix implementation designed to eliminate numerical differences between sampling and training during reinforcement learning for large language models on TPU. The company says its Zero-TIM system produced bitwise-identical action logprobs across separate execution paths, including in tests spanning 64 TPU v5p chips.

Key points
  • Google's Zero-TIM targets numerical differences between LLM sampling and training.
  • The contract compares sampler decode, serving prefill and trainer logprobs bit for bit.
  • The Tunix implementation uses Pallas kernels and explicitly ordered reductions.
  • Google verified the approach across 64 TPU v5p chips in a DP8×TP8 mesh.

The hidden numerical gap in on-policy learning

On-policy reinforcement learning requires actions to be generated and evaluated by the same policy. In large language model systems, however, rollout sampling and gradient updates commonly use separate execution paths.

The sampler generates tokens incrementally and records their behavior logprob. The trainer then evaluates those tokens in batched forward and backward passes. Google says the two paths can produce different logprobs even when they use identical weights. This creates a numerical gap that can compound across multi-turn trajectories, making training less stable or preventing convergence.

The cause is floating-point arithmetic. Addition is not associative, so changing the order of a reduction changes where rounding occurs. Google says the problem is particularly visible with bfloat16, and can arise in matrix products, normalization, tensor-parallel reductions and other model operations.

A three-way contract for matching logprobs

Google's zero train-inference mismatch, or Zero-TIM, requires sampler decoding, serving prefill and trainer forward passes to produce bitwise-identical logprobs for every sampled action. The comparison uses raw floating-point bytes under the same rollout weights, token history and sampling settings, with no rounding tolerance.

Serving prefill provides a diagnostic checkpoint between the sampler and trainer. Sampler-versus-prefill comparisons test whether incremental generation and full-sequence rescoring agree within the serving engine. Prefill-versus-trainer comparisons test whether serving and training agree across compilation and differentiation boundaries.

The implementation keeps the serving and training systems separate. Its recipe aligns within-shard kernel accumulation, explicitly orders cross-shard reductions, standardizes the action-logprob calculation and adds custom differentiation rules around serving kernels.

How Zero-TIM works on TPU

Within tensor-parallel ranks, Pallas kernels use fixed accumulation orders and explicit intermediate precision. Projection kernels accumulate partial products in float32 before casting outputs, while dedicated kernels handle RMSNorm reductions and SwiGLU intermediate casts. Google says the approach covers attention and MLP projections as well as the language-model head.

For row-parallel projections, communication is separated from addition. The implementation transports partial outputs with ppermute, then adds them in a fixed source-rank order rather than relying on the order selected by a collective implementation.

The sampler, prefill and trainer use matching sampling transforms and the same Pallas log-softmax construction to score the recorded action token. Sampled token IDs are retained across environment turns so each path evaluates the same token history. The trainer reuses the serving prefill computation for forward evaluation, while custom vector-Jacobian products supply backward rules.

Google also says long trajectories are processed in fixed-size chunks while carrying the paged KV cache forward. It used the implementation in Tunix and verified it across a multi-host 64 TPU v5p mesh with a DP8×TP8 configuration. The company presents the work as a reusable blueprint for teams building distributed JAX and TPU reinforcement-learning systems.

Related news

Google trains Gemma 4 for multi-turn agentic reinforcement learning
ResearchNovember 19, 2026

Google trains Gemma 4 for multi-turn agentic reinforcement learning

Google has demonstrated a multi-turn agentic reinforcement learning pipeline that trains Gemma 4 to use tools, respond to feedback and adapt across several steps. The…

Google publishes TPU guide for faster LLM and MoE workloads
For developersNovember 16, 2026

Google publishes TPU guide for faster LLM and MoE workloads

Google has launched the open-source AI-Hypercomputer/tpu-codesign guide, offering hardware-specific recommendations for designing and tuning models on its TPU systems.…

JetBrains asks 12 AI models for 1,200 programming hot takes
ResearchNovember 15, 2026, 18:02

JetBrains asks 12 AI models for 1,200 programming hot takes

Twelve AI models overwhelmingly attacked Python, Rust and TypeScript in a JetBrains experiment that collected 1,200 opinions about programming languages. The models also…