NVIDIA Details Full-Stack Approach to AI’s Growing Memory Pressure

Why AI workloads are pushing against memory limits

AI factories use a hierarchy that includes GPU high-bandwidth memory, CPU memory, local SSDs and shared storage. HBM is the fastest and most expensive tier, but model weights and KV cache compete for its limited capacity.

That pressure is increasing as frontier models reach trillions of parameters. KV cache stores the context needed for active requests, while agentic trajectories carry context across repeated model and tool calls. In some cases, those trajectories process up to 3 million tokens. Retrieval-augmented generation adds enterprise information to prompts, creating more context to process and retain.

NVIDIA says simply increasing HBM capacity does not solve the entire problem. Memory bandwidth determines how quickly data can move to the compute engines, and once that bandwidth reaches its limit, additional capacity may not be used effectively. The company describes this constraint as the memory wall.

NVIDIA’s strategy spans memory, storage and software

NVIDIA says its full-stack approach targets several sources of inefficiency. NVFP4 reduces the space required for model weights compared with FP8, and recent models from Black Forest Labs, Thinking Machines Lab, Reflection, Meta, StepFun and NVIDIA Nemotron 3.5 Super VL have been released with NVFP4 checkpoints.

Within an NVL72 system, weights can be distributed across up to 72 GPUs, leaving more HBM available for KV cache and concurrent users. NVIDIA also applies NVFP4 to KV cache to reduce its footprint. The company says NVIDIA Vera CPU can shorten tool execution and provide a nearby place for paused context through its large LPDDR memory, connected to NVIDIA Rubin GPUs over NVIDIA NVLink-C2C.

NVIDIA Dynamo inference software can move inactive context to NVIDIA CMX context memory storage, then return it before reuse. Dynamo’s KV Cache Reuse manages KV cache across GPU memory, CPU memory and storage, allowing context computed once to be reused across the system. NVIDIA says CMX partners include Cloudian, DDN, Dell Technologies, Everpure, HPE, IBM, NetApp, VAST and WEKA.

For retrieval-augmented generation, NVIDIA Nemotron Retriever models produce smaller embeddings, while the NVIDIA cuVS library supports building and serving the index across CPU and storage. The knowledge base therefore stays outside HBM, with only needed passages entering the model context.

Throughput gains are larger than HBM increases

NVIDIA points to throughput per dollar as a measure of how effectively its platforms use memory. The company says HBM capacity per GPU rose roughly 3% from NVIDIA Blackwell Ultra to the Vera Rubin platform, while agentic throughput per dollar rose by up to 30x on SemiAnalysis AgentX.

On the SemiAnalysis AgentX DeepSeek V4 Pro workload, NVIDIA says GB300 NVL72 delivers up to 10x higher throughput per dollar than the NVIDIA H200 platform at 100 tokens per second per user, with HBM capacity per GPU roughly doubling. At 125 tokens per second per user, Vera Rubin NVL72 delivers up to 30x higher throughput per dollar than GB300 NVL72, while HBM capacity per GPU rises by about 3%.

For AI factory operators, the stated goal is to produce more useful tokens within a power budget while meeting response-time requirements. NVIDIA’s argument is that coordinating precision, placement, reuse and movement across the memory hierarchy can lower token costs without treating additional HBM as the only answer to rising demand.