Google brings long-context training to TPUs through MaxText

Google has added long-context training support to MaxText, its open-source JAX training framework for Cloud TPU. The release supports four context-parallel strategies and, according to Google, can extend training contexts to 14 million tokens across TPU 5p and TPU 7.
- MaxText v0.2.4 adds long-context training support for Cloud TPU.
- The framework supports All-gather, Ulysses, Ring Attention, and USP.
- Google reports 70% MFU on TPU 5p at sequence lengths up to 4 million tokens.
- Context can be extended to 14 million tokens across TPU 5p and TPU 7.
- All four strategies use Tokamax Splash Attention.
Training models across millions of tokens
Google says long-context training now covers both pretraining and post-training for frontier AI models. Longer sequence lengths can give large language models the working memory to read an entire code repository, process hours of video, and reason through multi-step agentic tasks.
According to the Google Developers Blog, context windows have grown from 128K, 200K tokens to 1M, 2M tokens in three years, while models trained toward 10M tokens are arriving. Training at those lengths creates demanding system constraints. Attention computation grows quadratically with sequence length, while activations, including the keys and values associated with each token, can exhaust high-bandwidth memory when replicated across chips.
MaxText addresses this with context parallelism, which splits a long sequence across chips and streams keys and values instead of storing the full sequence on every device.
Four context-parallel strategies
MaxText supports All-gather, A2A, also known as DeepSpeed Ulysses, P2P, also known as Ring Attention, and the A2A plus P2P hybrid called Unified Sequence Parallelism, or USP. The strategies make different tradeoffs involving memory, communication, attention heads, and load balancing.
All-gather is the simplest approach, with each chip collecting keys and values from the others at every layer. Ulysses exchanges data across the token and attention-head dimensions, reducing the key and value memory held by each chip, but its scaling is limited by the model's head count. P2P sends key and value chunks around a ring, keeping only local and visiting chunks on each chip while overlapping communication with computation. USP combines an all-to-all operation on one axis of the chip mesh with a ring on another, allowing the context-parallel degree to extend beyond the head count.
All four strategies use Tokamax Splash Attention, Google's open-source block-sparse flash attention kernel. The kernel skips fully masked blocks and supports packed sequences without padding.
Reported TPU performance and availability
In Google's benchmarks, Llama 3.1 8B achieved 70% Model FLOPs Utilization on TPU 5p for sequence lengths up to 4 million tokens. Across TPU 5p and TPU 7, the context could be extended to 14 million tokens. Google also says TPU 7 maintained 50% MFU across all sequence scales below 6 million tokens.
The company identifies P2P as the primary strategy for very long-context training, while describing All-gather as the standard baseline when the key and value data fits in memory. On a fixed 64-chip TPU 5p slice using Llama 3.1 8B, Google compares the strategies across context lengths and says P2P can outperform All-gather across almost every context length when chunks and local batch size are sufficiently large.
The functionality is available today in MaxText v0.2.4. For users training models on Cloud TPU, the release provides several ways to distribute long sequences across chips while managing memory and communication. The reported results also show how Google is targeting the training bottlenecks that emerge as model context lengths reach millions of tokens.
Google for Developers: Announcing Long-Context Training on TPUs with MaxText Context Parallelism- Google Developers Blog


