Google makes GKE dynamic slicing generally available for TPU workloads

Separating TPU hardware from workload topology

Dynamic slicing decouples logical TPU slice formation from physical nodepool provisioning in Google Kubernetes Engine. Previously, teams using GKE standard had to manage separate, fixed nodepools for TPU slice topologies. Changing workload requirements could require those nodepools to be deleted and recreated, increasing startup time and operational complexity.

Google Cloud says the previous arrangement also created larger failure domains. A single hardware fault could bring down an entire slice while the infrastructure was reprovisioned. Dynamic slicing instead uses TPU optical circuit switching, or OCS, to connect TPU cubes into larger slices or divide them into smaller slices for multiple workloads.

Super-slicing and sub-slicing

GKE dynamic slicing is available in two forms. Super-slicing connects multiple TPU cubes to create slices larger than a single cube. Each cube consists of 16 virtual machines configured in a 4x4x4 slice topology. When a workload specifies its required topology in a Kubernetes workload YAML file, the GKE scheduler, or a custom scheduler, can request the corresponding slice at scheduling time.

Google gives a 4x4x16 super-slice as an example. It is formed by stitching together four Ironwood cubes, comprising 64 Ironwood VMs and 256 total chips.

Sub-slicing partitions one pre-provisioned TPU cube into smaller, independent slices. This allows multiple jobs with different topology requirements to run concurrently inside the same physical cube. When a job finishes, the scheduler deconstructs the slice and returns its partitions to the shared pool, without recreating the GKE nodepool.

Faster startup, recovery and capacity use

According to Google Cloud, dynamic slicing provides up to 5x faster job startup by assembling logical topologies from pre-provisioned TPU nodepools rather than waiting for a new nodepool deployment. It can also improve fleet utilization by packing smaller workloads into shared nodepools and using healthy hosts in partially healthy nodepools.

The company says recovery can be up to 4.5x faster because failures can be isolated to a TPU cube, described as a 16-VM 4x4x4 slice topology. Unhealthy cubes can be swapped in one to two minutes, while healthy partitions remain available for workloads.

Google Cloud says dynamic slicing can support training configurations ranging from large 8,000-chip single slices to multi-slices of multiple thousands of chips, as well as single-host four-chip configurations. Sub-slicing is also intended to support inference workloads from small to large-scale deployments.

What teams need to get started

Organizations must first pre-provision TPU capacity in nodepools of 16 VMs each. These units correspond to a TPU cube or sub-block and provide the baseline for creating custom slice topologies on demand.

Dynamic slicing introduces incremental provisioning nodepools. Google describes this as a fault-tolerant model that uses available healthy hosts even when some hosts in a 16-VM slice are faulty. The system continuously reconciles the nodepool and adds pending nodes when faulty hosts recover.

After enabling the slice controller and configuring the GKE scheduler, teams can request dynamic slices through workload annotations. GKE provides schedulers optimized for dynamic slicing, while customers can also use custom schedulers and health information to manage workload placement. This gives platform teams a way to share pre-provisioned TPU capacity across heterogeneous AI workloads while reducing the need for manual nodepool changes.