Pure KVA Keeps AI Agents Fast by Saving Evicted KV Cache, Pure Storage Says

Why agents strain inference servers

Pure Storage argues in a blog post that chatbots run on human time, while agents run on computer time. A person may take seconds or minutes to reply, which hides the cost of rebuilding a conversation's cache. An agent reads documents, calls a tool, reads the result and sends its next turn within seconds, and repeats that dozens of times per task.

According to the company, a rebuild that feels insignificant for a chatbot can add up to more than two minutes of waiting over a 60-turn agent task. The problem comes from limited GPU memory. An enterprise inference server may handle dozens or hundreds of conversations at once on a single node, and GPU memory cannot hold the KV cache for all of them. When memory fills, the engine evicts the caches of conversations not in active use, starting with the least recently used. An evicted agent that returns must have its cache rebuilt from scratch.

Pure says that with a handful of agents rebuilds are rare, but with a few dozen they happen on almost every turn, and the delay spreads to every agent on the server.

How Pure KVA works

Pure KVA is a KV cache backend for inference engines. It consists of a plugin in the engine, a service that runs beside it, and enterprise storage underneath. Pure says it is available now. It manages each conversation's cache across three tiers:

• GPU HBM: the high-bandwidth memory on the GPU package, holding conversations the engine is actively serving. A cache found here needs no rebuild.
• Host memory: DRAM on the same server, where evicted caches are kept one hop from the GPU for the fastest restores.
• NFS storage: far larger than a single server's memory and shared across the fleet, so a cache written on one node can be restored on another. Pure KVA also supports S3 object storage.

Eviction still happens with Pure KVA, and HBM stays the same size. What changes is the cost of coming back: the cache is read from host memory or storage rather than rebuilt on the GPU. In Pure's tests, a restore from NFS was up to 32 times faster than a rebuild and added no rebuild work to the GPU's queue.

The 40-agent load test

Pure built a test that adds one new agent every 15 seconds to a single Gemma-4 26B endpoint. No agent drops out, so all 40 run at once by the 10-minute mark. Each agent loads 10 documents on its first turn, giving a starting prompt of 22K to 33K tokens, then waits about 15 seconds between turns and resends the whole conversation each time. The company ran the same plan twice with the same random seed, once on vLLM with Pure KVA and once on the same engine without it. Each run lasted 750 seconds.

First turns were similar in both runs, with a median of 2.1 seconds with Pure KVA and 2.3 seconds without, since neither has a cache to reuse. For the first five and a half minutes, returning turns were also indistinguishable, at about 0.1 seconds, because every conversation still fit in HBM.

Then the run without KVA changed sharply. At 5.6 minutes, with about 22 agents running, its last returning turn was answered in under half a second. Thirty seconds later the median wait was 17 seconds, longer than the agents' own 15-second pause. It reached 31 seconds at eight minutes and about 45 seconds by the end, and never recovered.

With Pure KVA, the median stayed between 0.11 and 0.30 seconds once the first few agents were past their opening turns. The one rise to 0.30 seconds came at the 10-minute mark, when the last agents arrived, and it was back to 0.12 seconds within a minute. The slowest turn in the Pure KVA run took 4.6 seconds and was a first turn.

Pure says prompt growth was not the cause: inputs stayed between 21.8K and 34.1K tokens throughout. It attributes the jump entirely to HBM running out of room, after which every returning agent needed a rebuild and the rebuilds piled up faster than the GPU could handle them.

What it means for teams running agents

Pure points to the suddenness of the slowdown as a risk. A deployment that looks perfect with 15 agents, it says, can run more than 100 times slower at 40, so a small pilot can mislead. It also notes that rebuilds make the GPU recompute caches it already had, and that because agents wait for each answer before sending the next turn, slower answers mean fewer turns and fewer useful tokens.

On cost, the company estimates that matching Pure KVA's latency without a second cache tier would take roughly 1.8 times the GPU memory, which it says in practice means close to twice the GPUs. These figures come from Pure's own test on a single model and endpoint, so results on other setups may differ.