DigitalOcean Explains Batch Inference, With Up to 50% Lower Costs on OpenAI and Anthropic Models

What batch inference is and how it differs from real-time use

Batch inference, also called offline or asynchronous inference, means running an AI model over a large set of requests as a single job, with results returned later instead of immediately. DigitalOcean gives examples such as scoring an evaluation suite, tagging a year of product reviews, or generating synthetic training data.

The company argues that many jobs do not need an answer within seconds, yet are often run through the same real-time endpoints as live traffic, at live-traffic rates. It cites a Gartner forecast of $2.7 trillion in worldwide AI spending in 2026 and says enterprises are asking providers to help manage costs.

According to DigitalOcean, real-time inference answers one request at a time while a user or application waits, whereas batch inference processes thousands of requests in parallel. On its platform, batch results arrive within a 24-hour completion window, as a results file plus a separate error file kept for up to 30 days.

How the workflow runs on DigitalOcean

The company describes a five-step process that it says is common across most platforms:

• Collect inputs: prompts go in a JSON Lines (JSONL) file, one request per line, each with a unique ID.
• Submit as one job: you upload the file and set a completion window, which is 24 hours on DigitalOcean.
• Queue and run: the platform validates the file, queues the job and processes requests as capacity allows.
• Review output: results go to one file, and failed requests go to a separate error file.
• Download and use: you match each line to its input by ID and write the outputs to a database or pipeline.

DigitalOcean Batch Inference runs on the Inference Engine and currently supports OpenAI and Anthropic text models. Open-weight models are not supported at present.

The article also separates batch inference from continuous batching. The latter is a scheduling technique inside a model-serving engine that updates the active batch at every decoding step. Batch inference, by contrast, concerns how work is submitted. A provider can use continuous batching for either real-time or batch traffic.

Claimed benefits and early results

DigitalOcean lists several advantages. Batch requests cost up to 50% less than real-time rates, and customers are billed only for completed requests. Batch jobs use rate limits separate from serverless and dedicated inference, so a running job does not consume the real-time quota. The platform retries transient errors (429, 408 and 5xx) up to two times, and one failed request does not fail the whole batch. Jobs run at a lower scheduling priority on shared off-peak GPU capacity.

The company also notes that the discount is not identical everywhere. It points to a separate comparison across six providers, which it says shows differences in whether a flat 50% applies to every supported model and whether a 24-hour window is guaranteed.

For examples, the article cites developer Dumky de Wilde, who used OpenAI's Batch API to backfill about 60,000 job postings, extracting a summary, salary, technologies and job type from each. The full run cost about $10, and he reports that batches often finish in 10 to 15 minutes.

DigitalOcean engineers also ran a test on 250,000 real consumer complaints, labeling product line, sentiment and urgency, extracting entities and drafting summaries. All 250,000 requests completed across seven batch jobs, with the largest 40,000-request jobs finishing in 1.4 to 2.2 hours. The computed cost was $7.04, against $14.07 for the same tokens at serverless rates. Graded against the dataset's own labels, GPT-5 Nano agreed 83.6% of the time and Claude Haiku 4.5 reached 84.9%.

What it means for teams

Based on the source, batch inference fits high-volume, independent requests where results can wait, such as document extraction, ticket triage and evaluations. DigitalOcean says it does not suit interactive features or jobs that need results in minutes.

For teams already paying real-time rates for backlog work, the company's pitch is lower cost and a way to push bulk work through without crowding out the endpoints users depend on. The source also advises checking a sample of extracted fields against the original documents before loading a full output into a database.