NVIDIA outlines five ways to make every AI memory byte work harder
NVIDIA has outlined five co-designed technologies aimed at increasing the amount of useful AI work produced by each byte of memory. The company says the approach enables Vera Rubin NVL72 to deliver up to 30x the throughput per megawatt of GB300 NVL72 while keeping capacity at 288 GB per GPU.
- NVFP4 reduces model footprint by approximately 3.5x versus FP16.
- NVFP4 KV cache cuts cache footprint by up to 50% versus FP8.
- Wide expert parallelism spreads experts across eight or more GPUs.
- Dynamo routes requests using KV-cache location and worker load.
- NVIDIA says Vera Rubin NVL72 delivers up to 30x the throughput per megawatt of GB300 NVL72.
Memory efficiency becomes the focus
In a blog post dated Nov 05, 2026, NVIDIA describes high-bandwidth memory, or HBM, as the fastest, most expensive and most contested memory tier in an AI factory. The company argues that capacity alone does not explain the performance gains between generations. From GB300 to NVIDIA Rubin, capacity remained at 288 GB while bandwidth rose to 19.2 TB/s.
According to NVIDIA, the relevant question is how much useful work each byte delivers. Its design spans GPU architecture, scale-up interconnects, serving software, the CPU and storage, with the goal of reducing redundant weight copies, idle KV cache and repeated prefix computation.
NVFP4 and wide expert parallelism reduce pressure on HBM
NVIDIA says NVFP4 stores values using a 4-bit E2M1 element with two levels of scaling. Applied to weights, the format reduces model footprint by approximately 3.5x compared with FP16 and 1.8x compared with FP8. The freed memory can support more KV-cache capacity, larger batches and higher concurrency.
The company also says NVFP4 KV cache cuts the cache footprint by up to 50% compared with FP8, allowing approximately 2x more context to remain resident on the device. NVIDIA says NVFP4 has become the default precision for open agentic benchmarks including AA-AgentPerf, SemiAnalysis AgentX and MLPerf Endpoints.
Wide expert parallelism in NVIDIA TensorRT-LLM spreads mixture-of-experts weights across eight or more GPUs. NVIDIA says this leaves more HBM available for cache and increases arithmetic intensity through weight reuse. On GB200 NVL72, the company reports 130 TB/s of aggregate NVLink bandwidth, while large expert parallelism delivered up to 1.8x more output tokens per second per GPU than small expert parallelism in its cited test.
Dynamo treats KV cache as a shared resource
NVIDIA Dynamo is designed to prevent useful context from being recomputed simply because a request reaches a GPU that does not contain the relevant prefix. Its KV-aware router tracks where KV blocks reside and considers cached-prefix overlap alongside worker load when assigning requests.
The system also separates prefill and decode across different GPU pools. NVIDIA says this allows decode GPUs to dedicate more HBM to KV cache and batch size. Its KV Cache Runner tracks blocks across GPU, pinned CPU and storage tiers, including local SSDs and mounted network filesystems, so preserved prefixes can be restored after GPU eviction.
Together, these measures support NVIDIA's broader claim that the value of memory depends on placement and reuse as much as on capacity. For users of agentic AI systems, the intended result is more resident context, fewer repeated computations and greater concurrency. NVIDIA says the combined co-design delivers up to 30x the throughput per megawatt of GB300 NVL72 on the SemiAnalysis AgentX workload at 160 tokens per second per user.
NVIDIA Developer: Five Ways NVIDIA Co-Design Maximizes the Value of Every Byte in the AI Factory
