DigitalOcean makes prompt caching generally available for Serverless Inference

Why agents benefit from caching

AI agents never start from scratch. Each new call appends the result of the previous tool call to a context that already holds the system prompt, the tool definitions, every file the agent has read, and all earlier actions and results. According to DigitalOcean, more than 90% of the input tokens that agents send on Serverless Inference are tokens the model has already processed.

Prompt caching reduces the cost of those repeated tokens by up to 90%, depending on the model. The company notes that the actual discount and latency improvement vary by model and prompt.

Cache affinity routing

A cache only pays off if the request reaches the replica that holds it. Serverless traffic is spread across many replicas, so without special routing a repeat request can land on the wrong one, and the user pays full price for tokens already sent. DigitalOcean says that during the public preview, caching was on for everyone and the discount was real, but whether a request hit the cache was largely down to chance.

The GA release adds cache affinity. For each request, the router checks which replicas already hold most of the prompt, narrows the choice to that warm set, and then balances load across the least busy of those. If no replica is warm, it falls back to balancing across the whole pool. No session ID is needed, because a conversation's second message already contains its first, so the router recognizes the match itself.

In its October 2026 testing, DigitalOcean measured cache hit rates between 85 and 95% on its flagship open models. The company says cache affinity lifted hit rates from the mid-30% range into the 90%+ range. GLM 5.3 went from mid-single digits to 88 to 95%, while Kimi K3's benchmark prefix reuse ran from 84% on short prompts to 92%+ on long ones. DigitalOcean says these figures come from its own benchmark testing and that results will vary.

New controls, larger windows and a dashboard

GA adds several capabilities:

• prompt_cache_key: Caches are isolated between accounts by default. This optional parameter lets developers separate traffic inside their own account, for example keeping a legal team's cached context apart from a marketing team's, or one end customer's apart from another's. Requests with different keys do not share cache, and the key does not pin requests to a replica.
• Responses API support: Prompt caching now also works on the Responses API in addition to Chat Completions. Every response reports how many tokens were cached.
• Larger context windows: Context windows rise to 256K tokens and above, so more of a long codebase, document or conversation can be cached and reused as a single prefix.
• Dashboard: A new prompt caching dashboard in the control panel shows cache hit rate over time, a per-model breakdown and an estimate of dollars saved, helping users find prompts that are not benefiting from caching.

Getting started and what it means for users

Caching applies automatically to every eligible request, so there is nothing to switch on. Users open Serverless Inference in the DigitalOcean Control Panel, generate a model access key if they do not have one, and send requests to https://inference.do-ai.run/v1 as before. The prompt_cache_key parameter is only needed to group or separate traffic deliberately.

Cached tokens are billed at up to 90% off, varying by model, while uncached tokens are billed at the standard input rate. Each response states how many of each were used, so teams can check savings request by request. For teams running long-context agents on open models, the practical change is that the discount should now be applied far more consistently than in the preview. DigitalOcean says the feature is available today across its open model catalog, with coverage expanding over time.