High GPU Utilization Can Hide Costly Waste in On-Prem AI Inference
Enterprises moving AI inference onto their own hardware may find that GPUs running near 100% utilization still deliver poor economics, according to Everpure (the company behind Pure Storage). In a blog post, the company argues that the real measure is the effective price per token, and that cache reuse and smart routing decide it.
- Everpure says GPU utilization near 100% shows chips are busy, not that inference is efficient.
- It recommends tracking effective price per token: total cost inputs divided by tokens produced.
- Sharing KV cache across a fleet with NVIDIA Dynamo and Pure KVA can turn a roughly 10-second recompute into an approximately 500-millisecond injection, per Pure KVA benchmarks.
- A single gateway routes requests between on-prem and frontier models by difficulty and load.
- LangChain benchmarked NVIDIA's NeMo Switchyard at a 74% cost reduction through routing.
Why utilization is the wrong number
Everpure says the economics of inference have pushed more enterprises toward on-premises capacity over the past year. The company points to frontier output prices that had jumped somewhere between two and seven times, scarce GPUs and inference demand that was doubling every few months. The post describes the reference architecture for AI token optimization that Everpure announced at Everpure Accelerate London.
The problem, according to the post, is that on-prem costs are harder to see than cloud bills. There is no invoice, so the first warning usually comes from engineers who notice slow responses or a system that seems to need more hardware than it should. A dashboard showing GPU utilization near 100% looks like success, but it only shows that the chips are busy.
Everpure proposes a different metric: the effective price per token, meaning everything that went into producing tokens, such as GPU hours and power, divided by the tokens produced. The company says that with the wrong inference stack, a "free" open-weight model can cost as much per token as a frontier API.
Where GPU time goes: repeated prefill
The post explains that before a model produces any output, it must process the whole prompt in a step called prefill. Prefill scales quadratically with length, so doubling the prompt roughly quadruples the work. According to Everpure, a 100,000-token prompt on a large model can still burn 8 to 10 seconds of GPU time before the first output token exists, even on current hardware such as NVIDIA DGX B300 systems or NVIDIA Vera Rubin NVL72 systems.
Agentic AI makes this worse. The post gives the example of five hundred engineers launching coding agents against the same codebase, with the same system prompts, tool definitions and shared context. A naive setup rebuilds that context for every request, spending 4,000 to 5,000 GPU-seconds recomputing the same KV state. A round-robin load balancer can add to the waste, because the next request from a session may land on a node with a cold cache.
Frontier APIs price cache hits directly, with reused context costing a fraction of a fresh input token. On-prem, the cost appears instead as lost throughput: every GPU-second spent recomputing context is one not spent generating new tokens, which raises the cost of every token produced.
A fleet-wide cache and a single gateway
Everpure's proposed fix is to compute the key-value (KV) cache once and make it available across the whole fleet instead of keeping it on the node that built it. The post names NVIDIA Dynamo and Pure Key-Value Accelerator (KVA) as the infrastructure that moves KV state between nodes. In Pure KVA benchmark testing, according to the post, a roughly 10-second recomputation can become an approximately 500-millisecond cache injection.
The same design handles idle sessions. Agentic workflows often wait on tool calls, build processes or external APIs, so the system can demote an idle KV cache to shared storage and restore it without recomputation on any node with capacity. Everpure says this happens behind a single gateway and requires zero changes to client code.
That gateway also sits in front of both on-premises and external models. A lightweight difficulty estimator looks at what a prompt is asking and routes it to a suitable model. Overflow during traffic spikes can go to a frontier model rather than building queues on hardware that is already maxed out. In Everpure's framing, owned capacity is sized for steady-state load and rented capacity covers the peaks.
The post notes that the pattern is spreading. It cites NVIDIA's NeMo Switchyard, an open source routing layer that LangChain benchmarked at a 74% cost reduction by routing between an efficient open model and a frontier one, sending only a small fraction of calls to the frontier tier.
What it means for enterprises
For teams weighing on-prem inference, the post's message is that buying GPUs and running open models does not by itself lower token costs. Cache reuse, request routing and matching tasks to model size determine the outcome, and these are visible only if teams track the effective price per token rather than utilization alone. Everpure presents its reference architecture as one way to do that, combining on-prem capacity with frontier models when a task or a load spike requires it.
Pure Storage: The Architecture Behind More Efficient AI Inference
