Microsoft unveils Maia 300 and open datacenter specs for AI inference

Maia 300 targets inference efficiency

Microsoft says inference is becoming the dominant AI workload, which makes efficiency, latency and cost the key concerns. Azure Maia 300 is presented as a complete system for newer workloads: long-context reasoning, latency-sensitive conversational experiences, and sustained, multi-step agentic tasks.

According to Microsoft, Maia 300 is expected to deliver 40% more tokens per watt compared to leading GPU-based systems. In a full 96-accelerator configuration, the company says it delivers up to seven times the rack-scale token throughput of Maia 200.

Microsoft describes the design as a co-design effort that brings models, application frameworks, kernels, communication libraries, systems software and custom silicon into one optimization loop. Maia 200 is already deploying across the United States, Europe and Australia. Microsoft claims it delivers up to 1.7x more tokens per watt for built-in AI capabilities in Microsoft Teams, Copilot and Cowork experiences, compared with leading GPU infrastructure.

New CPU, data processing and security silicon

Microsoft argues that agentic workloads place new demands on CPUs, since each agent interaction generates orchestration, data retrieval, memory management and service calls. Its answer is Cobalt 200, described as an agent-native CPU. The company says it offers more than 50% performance improvement over Cobalt 100, reduces agent call latency by 33% and provides 23% higher throughput on agentic workloads.

The Azure Boost Data Processing Unit moves compression, networking transactions and security functions off the CPU onto purpose-built silicon. Microsoft says this delivers four times more performance while using significantly less power. Azure Integrated HSM, an in-house security chip, is included in all new datacenter servers and keeps cryptographic keys inside the hardware security boundary.

Microsoft, Marvell and Oracle are also introducing Tessera Integrated HSM, an open-source initiative that carries forward the Azure Integrated HSM architecture. It open-sources the architecture, specifications, protocols, drivers, software, firmware and silicon.

Networking, power and cooling

On networking, Microsoft and industry partners developed Multipath Reliable Connection (MRC), which rethinks Ethernet for AI. Its multi-plane architecture scales beyond 100,000 GPUs with two network tiers, and uses packet spraying to limit congestion. Microsoft also points to the Optical Compute Interconnect Multi-Source Agreement (OCI MSA), the first optical specification for multi-rack scale-up systems. The Ethernet for Scale-Up Networking (ESUN) workstream, with more than 175 organizations, has released specification version 1.0 and is developing version 1.1 for later this year.

For power, Microsoft is helping move the industry toward 800 volt direct current (VDC) infrastructure. Its Mt. Diablo architecture separates power conversion from the compute rack and supports approximately 100 kilowatts to beyond one megawatt. Specifications v0.7 and v1.0 have been contributed to OCP. The company is also working on solid-state transformer architectures, open requirements for backup systems that reduce reliance on diesel, and a common grid-interface specification for utilities, operators and vendors.

In cooling, the third-generation Azure Heat Exchanger Unit, Mt. Atlas, nearly doubles cooling capacity compared to the previous generation within the same operational footprint, according to Microsoft. Its next-generation AI datacenters will use a zonal cooling strategy with multiple independent water loops at different temperatures.

Fleet reliability and what it means

Microsoft has also published common RAS, debugging, telemetry and diagnostics specifications through OCP. It says it uses years of fleet data and AI to predict faults, pinpoint issues and recover more autonomously.

The announcement shows Microsoft pairing its own silicon with open specifications across the datacenter stack. Most performance figures are the company's own claims or expectations, and the source gives no pricing or availability details for Maia 300. For customers, the stated aim is AI that is more available, reliable and sustainable, while the OCP contributions let other vendors and operators adopt the same designs.