Why AI chip design is shifting toward memory, according to OXMIQ Labs CEO

From compute capability to data flow

SK hynix's series looks at how software, data center infrastructure, semiconductors and memory technologies work together in AI. Part 4 features Koduri, who has led GPU and AI accelerator architecture development at Apple, AMD and Intel. The article argues that as AI workloads move toward inference and agentic AI, computing power alone no longer defines system performance.

The key determinant of AI infrastructure performance is shifting from individual compute devices to data flow across the whole system. Performance and efficiency increasingly depend on how GPUs, memory, networks and storage work together. The article describes the move toward memory as a structural transition driven by physical limitations, not a temporary trend.

Following Moore's Law, the industry has packed more compute circuitry onto a single die, and today's AI accelerators have reached petaflops-level performance. Yet the bandwidth needed to feed those compute devices has not kept pace.

The energy and time cost of moving data

The article stresses that computation itself has become relatively inexpensive. Multiplication and addition require only a few femtojoules of energy. Reading data from on-chip SRAM consumes considerably more, and retrieving data from off-chip DRAM can cost hundreds or even thousands of times more. The farther data travels from the compute device, the greater the time and energy costs.

Memory capacity poses a similar problem. On-chip memory, from registers to shared caches, is fast but limited in capacity, while today's AI models handle far larger volumes of parameters and intermediate data. Data must be brought in from HBM or DDR, displaced after use and retrieved again, which adds to both time and energy burden.

The location of the bottleneck also depends on the workload. In large-scale training and large-batch inference, compute may appear to be the limit. In low-latency, small-batch inference, memory bandwidth can limit performance, even on the same silicon. That is why, according to the article, memory is increasingly a starting point for AI architecture design.

Transformers challenge the cache model

Computer architecture has long relied on caches, which assume data will be reused. For conventional workloads, even small amounts of on-chip SRAM could substantially cut DRAM accesses. The article says transformers, which dominate mainstream AI models, challenge that assumption because they process vast numbers of parameters and long contexts. It becomes difficult to keep all weights on-chip, and during inference each weight may be used only a limited number of times before new data is needed. Systems therefore repeatedly read from DRAM or HBM.

Longer contexts add to the load. The KV cache grows in proportion to context length, and attention operations increase both computation and memory accesses. Agents handling long contexts must retrieve state information at each stage of a task, and when many agents run at once, conventional memory architectures centered on DDR or GDDR can struggle with the data flow.

Techniques such as FlashAttention, PagedAttention and speculative decoding aim to reduce memory accesses and data movement. The article notes, however, that the physical cost of moving data cannot be eliminated entirely. Its conclusion for chip design is that data needs to be placed closer to the transistors that perform computation.

What it means for the AI hardware market

The article presents HBM as the first major architectural answer to the data supply problem, though the provided text breaks off before detailing it. The wider message, from a memory maker's perspective, is that AI semiconductor competition is shifting from raw compute speed toward reliably supplying enough data to keep compute resources continuously utilized. For chip designers and infrastructure builders, that means chip, package and memory architectures must all evolve together. The series continues with part 5, on physical intelligence and the role of memory.