Samsung Details zHBM, Memory Stacked Directly on the AI Processor

Why data movement is becoming the bottleneck

In the second part of its series on AI memory, Samsung argues that AI system performance depends increasingly on how quickly and efficiently data moves through the system, not only on compute capability. The company points to inference, reasoning, long-context processing and agentic AI as workloads that push in this direction. More generated tokens and larger KV caches place greater demands on memory, according to Samsung.

The challenge, as Samsung describes it, is to reach higher bandwidth while controlling the power and heat associated with data transfer. Conventional HBM architectures place memory beside the AI chip. zHBM moves beyond that layout by stacking memory on the processor itself, creating what Samsung calls shorter, denser connections.

Denser connections and shorter paths

A core technology behind zHBM is higher interconnect density between memory and compute. Samsung says it combines Wafer-on-Wafer (WoW) stacking with Hybrid Copper Bonding (HCB), which enables direct copper-to-copper connections at fine pitches. Reducing the size and pitch of Through Silicon Vias (TSVs) also allows more vertical connections within a given area.

Together with the Distributed I/O architecture, these connections increase data-transfer parallelism. More data can move simultaneously, which the company says enables high memory bandwidth without relying solely on increasingly high per-link data rates.

Direct stacking also shortens the distance data has to travel. Shorter interconnects can reduce driver requirements and lower the burden on interface circuitry. Samsung says it is exploring a short, wide interconnect architecture that reduces reliance on complex interface circuits and lowers the energy needed to transfer data, which becomes more important as bandwidth scales.

Thermal challenges and co-design

Placing memory and compute closer together creates new thermal problems. When memory sits directly above an AI chip and heat is dissipated upward, heat from the processor may have to pass through the memory stack. Samsung says it is exploring stack-height optimization that balances memory density with thermal performance. Stable operation requires the stack structure, interconnect density and thermal path to be designed together, the company says.

Because zHBM is tightly integrated with the AI chip, Samsung says memory and compute must be considered together from the early stages of development. Each AI chip has different requirements for performance, power, thermals and physical dimensions, so memory bandwidth, capacity, interconnect placement and stack configuration may vary by system architecture. Data interfaces, power delivery, stacking and thermal management need to be addressed together from the start.

What it means for customers and the market

Samsung says it brings together capabilities in memory, logic and foundry technologies, and advanced packaging. In its description, memory defines how data is stored and supplied, logic and foundry technologies support interface implementation, and advanced packaging enables the fine-pitch vertical connections and stacking structures needed for 3D integration.

The company aims to work with customers to develop zHBM solutions tailored to the performance, power, thermal and architectural requirements of their AI chips, with the goal of jointly optimizing memory-to-compute connectivity within the actual AI system. The article gives no performance figures, release timing or availability, and zHBM remains under development. For chip designers, the message is that future memory choices may be made together with the processor design rather than separately.