Microsoft details Maia 300: 96 accelerators per rack and a 12,288-chip superpod

October 14, 2026, 00:08Source: Microsoft

Microsoft has published an architecture deep dive on Maia 300, the third generation of its Azure inference accelerator, built to maximize useful tokens per watt and per dollar for agentic workloads. According to Microsoft, the rack doubles to 96 accelerators and the scale-up domain grows to a superpod of 12,288 accelerators.

Key points

Maia 300 is Microsoft's third-generation Azure inference accelerator, built for agentic workloads.
Each accelerator has eight HBM4 stacks with 288 GB and 18 TB/s, which Microsoft says is 2.5 times the bandwidth of Maia 200.
Racks double to 96 accelerators within 167 kW, and a superpod scales to 12,288 accelerators.
Compute is on TSMC N2P, and Microsoft claims a 36% rise in compute FLOPs.
Microsoft claims up to 2.5x more performance per watt than Maia 200 at rack level.

A system built around inference

Microsoft says Azure Maia has been designed from the start around one idea: optimize the whole system for LLM inference so that every watt and every dollar delivers more intelligence. Maia 300 is described as the fullest expression of that approach, with software, silicon, rack infrastructure, networking and datacenters co-designed to work as a single platform.

The company argues that agentic workloads, including multi-step execution, long-context decoding and tool use, change the computational profile and arithmetic intensity of inference. Meeting that demand, it says, requires sustaining locality, supplying compute with enough memory capacity and bandwidth, minimizing data movement, scaling communication efficiently and fitting as much capability as possible into a datacenter's fixed power and cooling envelope. Microsoft points to its own products, from Microsoft Copilot and GitHub Copilot to Microsoft Foundry, as the source of its view into these workloads.

Chip design: chiplets, NUMA memory and narrow precision

Maia 200 introduced the Software Defined Locally Accessed Dataflow Architecture, or SDLA, which separates control path programming from data path execution and lets software explicitly coordinate data placement, movement, conversion and synchronization. Maia 300 evolves this design for long-context, reasoning-intensive and decode-heavy inference.

Multi-chiplet layout: compute sits on TSMC N2P, while memory-controller and I/O functions go onto separate dies optimized for bandwidth, connectivity and cost. Microsoft says this raises system compute FLOPs by 36%, adds 2.5 times HBM bandwidth and provides 14% more scale-up bandwidth.
Low precision: enhanced MXFP4 and sub-3-bit storage formats reduce memory footprint and data movement, allowing larger models, longer contexts and more concurrent sessions within the same power envelope.
Kernel support: the vector engine adds optimizations for inference kernels such as softmax and low precision data casting.
NUMA memory: memory moves from a uniform to a non-uniform organization across chiplets, so data sits next to the compute that uses it and travels shorter distances.
Memory hierarchy: software-managed on-die SRAM combines with eight HBM4 stacks, giving 288 GB of capacity and 18 TB/s of bandwidth per accelerator.

Compared with Maia 200, Microsoft says this means 2.5 times the memory bandwidth, 33% more capacity and approximately 2 times the HBM bandwidth per watt.

One scale-up fabric from pod to superpod

Maia 300 treats the network as an extension of its dataflow architecture. Its AI network controller implements a custom AI transport based on Ethernet, and a two-tier network builds a superpod of up to 12,288 accelerators.

In the rack, each accelerator splits its scale-up connectivity between direct links and switched paths. Direct links connect two accelerators in a tray, while the switched fabric connects all 96 accelerators in the rack. In-rack links are electrical to cut power and cost. Second-tier switches use optics to connect up to 128 racks into a superpod.

Single communication model: the same semantics apply inside the pod and across the superpod, and accelerators can start communication with any other accelerator in the superpod without involving the host or a discrete scale-out NIC.
Three latency domains: a directly connected pair is lowest, accelerators in a pod are one switch hop away, and accelerators across pods are three switch hops away. Software can keep latency-sensitive tensor-parallel traffic local.
Adjustable bandwidth: the split between in-rack and cross-rack bandwidth is a parameter set by job packing, and an accelerator can drive up to 750 GB/s to any peer in the superpod when only some accelerators go cross-rack.
Flexible placement: tensor parallelism can stay within a 96-accelerator rack while expert and pipeline parallelism span racks. A 64-accelerator job can run as four 16-accelerator shards in four racks.
Disaggregated inference: prefill and decode workers can be sized and placed independently, with KV state moving directly to decode workers over the same fabric.
Resilience: schedulers can use hot-spare capacity in another rack transparently to software kernels, trading extra path latency for availability.

Rack design and what it means

At datacenter scale, Microsoft says efficiency depends on how much useful infrastructure can be deployed within a fixed power and cooling limit. The Maia 300 rack co-designs server layout and density, network topology and switching, power and cooling, and advanced telemetry, and Microsoft claims up to 2.5x more performance per watt than Maia 200. The rack holds 96 accelerators across 24 four-accelerator trays within 167 kW, doubling the accelerator density of Maia 200.

For Microsoft's customers, the stated aim is more capacity, capability and performance in products such as Copilot, GitHub Copilot and Foundry. The figures are the company's own, and the source text does not give pricing or availability details.