CoreWeave reports 51-52% MFU and 3.66-day MTTF in 1,024-GPU H100 test

The problem: interruptions and low utilization

CoreWeave opens with the claim that production training runs fail not just from a lack of compute but from the compound effect of interruptions and inefficiencies. It points to a paper called "Revisiting Reliability" from a leading AI lab, which reported an MTTF of 0.33 days at 1,024 GPUs. That means a crash roughly every eight hours, with days lost to reloads and wasted steps.

The company also notes that MFU often falls to 35-45%, which it says cuts effective speed in half. It cites a separate training report from a leading AI lab that documented hundreds of unplanned interruptions over 54 days at 16K GPUs as evidence of the challenge.

How the study was set up

According to CoreWeave, the test used production-scale training rather than synthetic workloads. The parameters were:

• Model: a 30-billion parameter Llama 3-style architecture
• Scale: up to 1,024 NVIDIA H100 GPUs
• Dataset: 2 trillion tokens from Dolma v1.6
• Framework: production Megatron-LM
• Metrics: MTTF, ETTR, MFU, checkpoint performance and tokenization throughput

On the hardware side, CoreWeave used bare-metal H100 clusters to remove hypervisor overhead, with NUMA pinning under its control. It describes a dual-fabric design: NVIDIA Quantum InfiniBand for all-reduce operations and a separate Ethernet network offloaded by NVIDIA BlueField DPUs for storage traffic, to prevent network contention.

For orchestration, the company uses SUNK (Slurm on Kubernetes), with topology-aware scheduling and health probes that evict failing nodes before they affect jobs. Failed processes are re-queued automatically and restart in about 90 seconds, compared with more than four minutes of manual triage, according to CoreWeave.

For storage, Tensorizer-based asynchronous checkpointing cut save time from 129 seconds to 17 seconds while keeping compute utilization above 99%. A custom gpt_bpe tokenizer reached 63M tokens per second, which CoreWeave says is 6-12x faster than HuggingFace Tokenizers.

Reported results

CoreWeave reports the following outcomes:

• 51-52% MFU on H100 GPUs, versus 35-45% typically reported
• 3.66 days MTTF at 1,024 GPUs, a 10x improvement over the 0.33-day baseline
• 97.5% ETTR (Effective Training Time Ratio)
• 8x faster checkpointing through the asynchronous Tensorizer implementation
• A projected 43.7% MTTF improvement at 16,384 GPUs compared with Llama 3's reported numbers

The company also compared its figures with published configurations. Against another AI research group, it cites 51.9% MFU versus 40.43% (a 28% improvement). Against another leading AI lab, it cites 49.2% versus 41.85% (an 18% improvement). CoreWeave says it also reached performance and reliability parity with NVIDIA DGX Cloud Benchmarking recipes.

As an example of the automation, CoreWeave describes a 512-GPU run at 2:17 a.m. in which SUNK detected a failing node, evicted it, rescheduled the workload and resumed training within 3 minutes, before the on-call engineer was alerted.

Cost and what comes next

CoreWeave frames the gains in money terms. For a 30-day, 1,024-GPU run at $2.10 per GPU-hour, raising average MFU from 42% to 51% yields roughly 6,000 GPU-hours of additional effective compute, worth $12,600, without changing the invoice. The company says the 9 percentage point gain means training finishes 18% faster, saving nearly a week on month-long jobs.

These are the company's own measurements and calculations, and the full methodology sits in a 30-page technical report that CoreWeave says is available now, including survival model mathematics and raw logs. The company adds that it is already applying the optimizations to NVIDIA GB200 NVL72 clusters and building live MFU dashboards for customer workloads.

For teams planning large training runs, the message is that infrastructure design affects both speed and stability at thousand-GPU scale. Readers who want to rely on the figures will need to check the technical report against their own workloads.