CoreWeave Claims Inference Autoscaling 8 to 10x Faster Than a Major Generalized Cloud

What CoreWeave tested

CoreWeave ran benchmarks comparing its inference service with one of the most popular generalized cloud providers, which it does not name. The tests covered container spin-up time, networking speed and overall latency for autoscaling, all of which affect inference performance, according to the company.

The model used was EleutherAI's GPT-J-6B. CoreWeave chose it because it is a standard mid-sized model with 6B parameters that fits on a broad range of GPUs. The company also notes that it is not image-based, which it says makes it suitable for benchmarking LLM-based machine learning inference.

The results, according to CoreWeave

The company reports three main findings:

• Container spin-up: CoreWeave averaged roughly 30-45 seconds, while the generalized cloud averaged roughly 120-150 seconds. CoreWeave describes this as a 3-5x improvement.
• Object storage transfer: CoreWeave's average transfer speed was 700MB/s, against 72MB/s for the generalized cloud's storage in the same region. CoreWeave calls this 3-5x faster and says speeds can reach 12Gb/s in some instances.
• Full autoscaling latency: On CoreWeave the average was 45-70 seconds, made up of 30-45 seconds for container spin-up and 15-25 seconds to load the model from accelerated object storage. On the generalized cloud it was 270-390 seconds, from 120-150 seconds of spin-up plus 150-240 seconds of model loading.

From the full autoscaling figures, CoreWeave concludes that serving ML inference on its platform is 8-10x faster. These are the company's own tests, and the comparison provider is not identified.

Why the architecture matters, in CoreWeave's view

CoreWeave argues the difference comes down to the underlying infrastructure. It says most generalized cloud providers offer managed Kubernetes that runs through a hypervisor, a program that manages virtual machines. Because that layer rests on top of the operating system, CoreWeave says, it cannot communicate directly with the hardware, which can reduce efficiency.

CoreWeave's stack is built around bare-metal compute. The Kubernetes layer sits almost directly on the hardware, with only a lightweight OS required. The company calls this serverless Kubernetes. It also offers the option to host virtual machines inside Kubernetes containers. For traditional cloud setups, CoreWeave says spinning up new instances can take more than 5-10 minutes.

The company also points to process differences. On a generalized cloud, it says, users must first have enough compute quota, and raising a quota can require requests and discussion of plans and budget, which it describes as especially restrictive for smaller customers. Pods are typically spun up on top of virtual machines, and with a standard container concurrency of 1 each pod is assigned to a VM, adding delay. CoreWeave also says the KNative activator may be unable to queue all incoming requests during scaling because of buffer queue limits, so some requests time out.

On CoreWeave, it says, pods can be started without launching a virtual machine, and users can autoscale across tens to thousands of GPUs on demand and scale back down. It also uses Tensorizer, an open-source tool for quickly loading a PyTorch model from HTTP/HTTPS and S3 endpoints.

What it means for users

CoreWeave frames the results around user experience: faster autoscaling means faster responses and better access to on-demand compute during traffic spikes. It argues that general-purpose clouds are less suited to the demands of unpredictable inference workloads, and says infrastructure choices also affect price.

For teams serving models to users, the takeaway from the source is that scale-up time and model loading speed can dominate responsiveness during bursts. Since the figures come from a vendor's own tests on a single mid-sized model against an unnamed competitor, teams may want to run their own comparisons with their workloads.