Google publishes TPU guide for faster LLM and MoE workloads

Google has launched the open-source AI-Hypercomputer/tpu-codesign guide, offering hardware-specific recommendations for designing and tuning models on its TPU systems. The guide targets large language models, Mixture-of-Experts models and other hardware-intensive workloads.
- Google launched the open-source AI-Hypercomputer/tpu-codesign guide.
- The guide recommends aligning key model dimensions to 256 for TPU systolic arrays.
- LatentMoE compresses routed tokens before expert dispatch.
- Google measured 2.8× lower permute time in a 30B, 128-expert MoE test.
- The recommendations cover Ironwood, TPU 8t, TPU 8i, TPU 8tx and TPU 8i Max.
Design models around TPU architecture
Google says TPUs are built for machine learning workloads, with wide systolic arrays, dedicated SparseCore engines and custom interconnect fabrics. Those features can deliver strong performance per total cost of ownership, but they also create architectural constraints that developers must account for when designing models.
The guide covers training and inference on Ironwood, identified in the post as TPU v7x, as well as future TPU generations. Google divides its recommendations into three groups: shape rules, architecture choices and ways to express parallelism. Shape rules are presented as practices developers should adopt, while architecture choices should be tested through ablations because they can involve model-quality tradeoffs.
Shape alignment is a central recommendation
One of the guide's main rules is to align reduction and output dimensions to 256, matching the dimensions of TPU's 256×256 MXU systolic array. Google specifically highlights head_dim, saying that values of 64 or 128 can leave the MXU at least 50% idle during flash-attention QK products.
The guide recommends using head_dim = 256 with proportionally fewer query and key-value heads, while preserving total projection widths and logical KV cache bytes. According to Google's measured Ironwood profiling, reshaping Qwen3-30B MoE attention from 32 × 128 to 16 × 256, with identical parameters and FLOPs, improved end-to-end training speed, with larger gains at longer context.
Google also recommends static shapes because XLA compiles each program for exact tensor shapes. For dynamic workloads, it describes using buckets, masking and packed training sequences, while keeping buffers fixed and representing changing data extents with small integer arrays.
LatentMoE can reduce routed-token traffic
Another highlighted design choice is LatentMoE. Instead of dispatching every routed token at the full d_model width, LatentMoE projects tokens into a narrower latent width before sending them to experts, then projects them back after combining the results.
Google says this approach can reduce expert weight reads and all-to-all dispatch traffic. Its TPU-specific recommendations include routing on the full d_model before compression, keeping the latent width a multiple of 256, using FP8 for dispatch and BF16 for the lightweight projections, and placing expert parallelism on one short 3D torus axis.
In Google's measured 30B, 128-expert MoE test, a latent width of 768, described as 2.67× compression, cut on-device permute time by 2.8× and increased end-to-end training throughput by 14% in tokens per second. The reported comparison was 322 versus 270 TFLOP/s per chip, and the compressed latent representation freed about 25 GB of activation HBM per device.
For users adapting an existing model, Google says the guide's parallelism section can be applied without changing the checkpoint. The company says the broader rules also apply to dense LLMs, multimodal models and world models, while the accompanying repository will add model- and workload-specific material over time.
Google for Developers: TPU Model Design Considerations and Optimization Guide- Google Developers Blog


