CoreWeave claims up to 20% higher GPU cluster efficiency than public benchmarks
CoreWeave says its AI cloud platform achieves Model FLOPS Utilization (MFU) above 50% on NVIDIA Hopper GPUs, which the company describes as up to 20% higher performance than public foundation model training benchmarks. According to CoreWeave, those benchmarks typically range from 35% to 45%.
- CoreWeave estimates MFU above 50% on NVIDIA Hopper GPUs, versus 35% to 45% in public benchmarks.
- On 128 H100 GPUs it reports 51.9% MFU against Aleph Alpha's 40.43% and 49.2% against MosaicML's 41.85%.
- Preliminary tests with NVIDIA DGX Cloud Benchmarking Recipes show performance on par with NVIDIA's reference system.
- CoreWeave cites a goodput rate as high as 96% and storage speeds up to 2 GB per second per GPU.
- The results are the company's own analysis, not independent verification.
What the efficiency gap means
MFU measures observed throughput against the theoretical maximum a system could deliver at peak FLOPs. CoreWeave argues that the complexity of running AI infrastructure causes a majority of GPU compute capacity to be lost to system inefficiencies, with empirical evidence suggesting performance levels of 35% to 45%.
The company calls the distance between that range and the theoretical 100% the "AI Efficiency Gap". According to CoreWeave, closing it can improve performance and model quality, speed up development timelines and reduce AI model costs.
CoreWeave also contends that general-purpose clouds run by traditional hyperscalers were created over a decade ago for uses such as search, e-commerce, web hosting and databases, and rely on CPU-based web-scale compute. In its view, AI labs and enterprises need a specialized GPU-based environment built for AI workloads.
How the comparison was run
CoreWeave launched Llama 3.1 training runs on a 128 NVIDIA H100 GPU cluster (16 nodes) and compared the results with published figures from Aleph Alpha and MosaicML.
- Aleph Alpha and the Hasso Platner Institute published an MFU of 40.43% for a Llama 30B model with an 8K token context window on 128 NVIDIA A100 GPUs. With a dense 30B parameter model and the same parallelism hyperparameters, CoreWeave reports 51.9% MFU on 128 NVIDIA H100 GPUs, an increase of 28%.
- MosaicML achieved 41.85% MFU training its 30B MPT model with a 2K token context window on 128 NVIDIA H100 GPUs. CoreWeave reports 49.2% for a model of the same size, context window size, micro-batch size and parallelism hyperparameters, an 18% increase.
CoreWeave acknowledges that the first comparison involves different GPU types, but says the results remain comparable because MFU is a percentage of the accelerators' theoretical FLOPs rather than an absolute score.
The company also ran NVIDIA's DGX Cloud Benchmarking Recipes on its H100 instances. It says preliminary testing shows its infrastructure is equally performant to NVIDIA's highly optimized reference system, and that results are on par across all tested training and fine-tuning workloads, for both BF16 and FP8.
The stack behind the numbers
CoreWeave attributes the results to optimization at every layer of its platform. The company lists these components:
- Networking with NVIDIA Quantum InfiniBand and NVIDIA Spectrum switches, which it says reduces NCCL communication overhead that typically dominates distributed training time.
- Cluster validation, health monitoring, proactive node replacement and deep observability, which CoreWeave says yield a goodput rate as high as 96%.
- CoreWeave Kubernetes Service (CKS), which runs Kubernetes directly on bare-metal servers.
- CoreWeave AI Object Storage, with data transfer speeds as high as 2 GB per second per GPU across hundreds of thousands of GPUs.
- Slurm on Kubernetes (SUNK), which runs Slurm-based workloads on more than 32K GPUs with topology-aware scheduling.
- CoreWeave Tensorizer, which the company says enables asynchronous checkpointing and virtually eliminates its impact on MFU.
- Managed Grafana with pre-built dashboards covering InfiniBand bandwidth, GPU temperature, power consumption and real-time alerts.
What it means for customers
Alexis Bjorlin, VP of NVIDIA DGX Cloud, said: "CoreWeave demonstrates performance excellence on par with our NVIDIA H100 cloud reference architecture." CoreWeave adds that it continues to work with NVIDIA following the launch of its NVIDIA GB200 NVL72-based instances, aiming to move observed performance closer to theoretical performance.
The company says the findings match feedback from other customers, who report higher performance along with better resilience and usability. It names Cohere, Mistral and IBM as customers choosing CoreWeave as their AI cloud provider.
The figures come from CoreWeave's own analysis and testing, and the company describes the NVIDIA reference comparison as preliminary. For teams weighing GPU cloud options, the claims suggest MFU and goodput are metrics worth asking providers about.
