CoreWeave joins Red Hat's llm-d open source project as a founding member
CoreWeave has become a founding member of Red Hat's new llm-d open source project, alongside IBM Research, Google and NVIDIA. According to CoreWeave, the project aims to bring vLLM to production-scale AI inference using Kubernetes.
- CoreWeave is a founding member of Red Hat's llm-d open source project, with IBM Research, Google and NVIDIA.
- llm-d builds on vLLM and Kubernetes to scale inference beyond a single server.
- Features include Prefill and Decode Disaggregation, KV cache offloading, an AI Inference Gateway and NIXL-based routing.
- CoreWeave says its Tensorizer contribution to vLLM enables more than 5X faster model loading than HuggingFace when scaling from zero.
- CoreWeave plans to contribute Kubernetes operators, benchmarking harnesses and monitoring tools.
What llm-d is meant to do
llm-d extends vLLM beyond the limits of a single server, according to CoreWeave's announcement. The source describes vLLM as the open source de facto inference server, with day 0 model support for emerging frontier models and support for a broad list of GPUs and accelerators.
CoreWeave argues that problems appear as foundation models grow in size, evolve in capability and increasingly support agentic applications. Developers then have to deploy these models at scale while managing infrastructure, costs and latencies for a wide range of use cases. The company says this creates a need for open standards and broader industry collaboration, including interoperability for inference workloads across platforms.
To address this, llm-d uses Kubernetes orchestration and adds several inference capabilities, which CoreWeave says deliver greater performance and lower latency:
- Prefill and Decode Disaggregation, which lets these components scale independently
- lmcache-based KV cache offloading to optimize memory use
- AI Inference Gateway
- AI-aware network routing for more efficient data transfers using NVIDIA Inference Xfer Library (NIXL)
Kubernetes-powered clusters and controllers also handle scheduling of compute and storage resources as workload demand fluctuates, and support interoperability across cloud platforms.
CoreWeave's planned contributions
CoreWeave says its purpose-built AI cloud platform was developed from the ground up on Kubernetes, and that it has previously made open source contributions such as CoreWeave Tensorizer. It plans to bring its experience from managing large-scale Kubernetes and AI inference deployments to llm-d, with the stated goal of reducing the heavy lifting for everyday developers.
The company lists several areas of contribution: Kubernetes native operators for deployment and management, testing and benchmarking harnesses for production workloads, and tools for deploying and monitoring the inference stack at scale across multiple GPUs and clusters. It also says it is focused on reducing latency and optimizing costs.
CoreWeave states that it contributed CoreWeave Tensorizer to vLLM with expanded support for llm-d. According to the company, this enables more than 5X faster model loading compared to HuggingFace when scaling from zero, using a "zero-copy" approach. The figure is CoreWeave's own claim, and the source gives no further benchmark details.
The company also says that, as a cloud platform that is first to deploy the latest hardware, including GB200, it will lead efforts to unlock the full potential of those hardware advances.
What it means for developers
CoreWeave says the combined capabilities should raise developer productivity and make it easier to build once and run across different cloud platforms that support Kubernetes deployments. The company also notes that the project has already drawn support from gen AI model providers, AI accelerator makers, AI cloud platforms and the developer community.
CoreWeave invites both established enterprises and startups to take part, describing llm-d as a flexible platform to build on. The announcement does not give release dates, pricing or availability details for the project, so how quickly these features reach production use remains to be seen.
In the announcement, CoreWeave also notes that SemiAnalysis's ClusterMAX Rating System recognized it as the only cloud provider to earn the top Platinum tier rating.
