Google makes Pathways generally available for large-model training

A response to the limits of SPMD training

Most large-model training uses the Single Program, Multiple Data, or SPMD, model. In this approach, every accelerator runs the same program in lockstep and coordinates with collective operations such as AllReduce. Google says this works well when a job is uniform, but becomes harder to manage when different parts of a computation need to run at the same time.

The company points to pipeline parallelism and reinforcement learning post-training as examples. In a pipeline, devices run different stages and can spend time waiting for activations or gradients. In an RL loop, a sampler, reward model and trainer are separate programs that need to overlap. SPMD makes these dependencies harder to coordinate because each process has visibility into only its local devices and data.

One client describes the whole computation

Pathways takes a different approach by representing an end-to-end computation as a graph. The blocks of work form nodes, tensors moving between them form edges, and devices provide the locations where the nodes run.

With Pathways, developers define that computation from a single Python client without writing custom coordination code. The client runs on an ordinary CPU machine and works with virtual devices rather than directly managing physical chips. According to Google, a resource manager maps those virtual devices onto real accelerators, while workers on accelerator hosts run the compiled programs they receive.

This design separates the description of the computation from the hardware on which it runs. Google says the runtime handles transfers, dependencies and ordering along the graph, while the virtual-device model allows the devices used for a computation to be changed.

Designed for large, changing workloads

Google says large training runs also need to continue through hardware failures. During Gemini 2.5 training, hardware failures interrupted runs multiple times per hour, according to the company. In an SPMD setup, one lost machine can prevent a collective from completing, stop the job and force a restart from the last checkpoint.

The company also describes limits around resource ownership. In the SPMD model, processes keep their accelerators for the duration of a run, even when one stage needs fewer chips than another. Devices cannot be added or returned during execution, and separate jobs require separate allocations and processes.

Pathways is now available on Google Cloud and runs JAX programs on TPUs. For teams training large models, the announcement offers Google's own coordination system as a way to describe more complex workloads centrally instead of managing every process and data dependency separately.