llm-d 1.0 arrives with production features for open AI inference at scale

What llm-d 1.0 is meant to solve

llm-d was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, and is now governed by the Cloud Native Computing Foundation (CNCF). The contributors announced version 1.0 on October 20, 2026.

The pitch rests on a cost problem. According to the announcement, many organizations are discovering that running AI at scale can be as challenging and expensive as building the models themselves. Agentic workloads add to the strain: they continually access and reuse vast amounts of context and run parallel agents that can generate unpredictable bursts of activity. The announcement says managing that context often consumes more resources than generating the final output.

llm-d already offers prefix-cache and load-aware routing, KV cache offloading, prefill/decode disaggregation, wide expert parallelism and batch and async inference. It works across multiple model servers, including vLLM and SGLang. Version 1.0 is designed to bring this to what the project calls Day 2 operations, the ongoing work of running inference in production.

New capabilities in the 1.0 release

The project lists several additions:

• Multi-tenant quality of service: priority and fairness-aware flow control lets latency-sensitive requests and bursty agent workloads share a fleet without starving one another.
• Zero-downtime operations: router high availability with hot standby and rolling updates keeps traffic flowing during upgrades and planned maintenance.
• Prefill/decode management: replicas are handled as one logical unit, with coordinated rollouts and lockstep upgrades, which the project says makes disaggregated serving practical in production.
• A stable foundation: a v1 configuration API and a hardened request path, validated across clouds and accelerators, give adopters a release they can standardize on.
• Agent-aware serving: session-aware routing keeps multi-turn conversations close to their cached context, while turn-priority scheduling and native Responses API state support help agents reuse context and reduce recomputation.

Reported results and ecosystem

The project says it has more than 500 contributors from over 25 organizations. It describes llm-d as broadly adopted by frontier model labs, hyperscalers, neoclouds, hardware accelerator makers and major enterprises, and cites demonstrations with up to five to 10 times cost reductions for context-heavy agentic workloads and three times greater output throughput.

In one use case, open frontier models and llm-d enabled IBM Research to explore quantum error correction hardware at 11.5 times lower inference cost than an all-proprietary configuration. Researchers reported a prefix-cache hit rate of approximately 98% and a blended serving cost of about $0.11 per million tokens.

In a separate benchmark demonstration, researchers used llm-d to deploy GLM 5.2, an approximately 753-billion-parameter open-weight model, on widely deployed NVIDIA GPUs. On a workload with hundreds of agentic sessions running in parallel, the deployment delivered over 6.6 million output tokens per minute at peak, serving up to 3,000 concurrent coding agents with zero preemptions. At current cloud rental rates, the announcement says, costs would have been 5 to 10 times less per token than equivalent commercial API pricing, with the largest savings on input-heavy traffic typical of agentic workloads.

What it means for enterprises

For organizations moving AI from experiments to production, the release positions llm-d as an open, standardized layer for distributed inference. The GLM 5.2 demonstration is framed by the project as a way for enterprises to use existing GPU fleets to serve frontier-class AI at scale.

Google Cloud's Mark Lohmeyer said the company is integrating llm-d 1.0 into Google Kubernetes Engine and enabling it across its Google TPU and NVIDIA GPU portfolio. Red Hat's Brian Stevens described the release as "a standardized, hardened core for distributed inference." The community says it will keep adapting the software to new hardware and models as they emerge.

The performance and cost figures come from the project's own demonstrations and benchmark workloads, so results in other environments may differ.