CoreWeave: Distributed File Storage Sustains 1 GiB/s per GPU in Tests

What the I/O pattern of large-scale training looks like

CoreWeave begins with a representative workload: training a multi-billion parameter model across 4096 NVIDIA H100 GPUs. According to the company, the first two hours of training show two distinct patterns.

• Initial data loading: the model loads data into GPU memory at startup, causing an intense burst of high-volume, partially random reads. CoreWeave notes that part of optimizing data is storing it to minimize random reads and maximize sequential reads.
• Checkpoint operations: periodic write spikes occur when model state is written to storage. The example uses synchronous writes, which create distinct I/O peaks, although some frameworks support asynchronous checkpointing.

The workload shows peak read rates of approximately 70 GiB/s and write spikes reaching 50 GiB/s. CoreWeave says checkpoint write rates are limited by the number of ranks writing checkpoints and the connectivity of those hosts, and that these metrics typically scale with GPU count. The company adds that the storage infrastructure supporting the cluster keeps substantial headroom above these observed peaks.

Architecture built on VAST Data

CoreWeave Distributed File Storage uses VAST Data's storage architecture, a disaggregated two-tier system. CNodes (Controller Nodes) handle data management, I/O scheduling and storage operations. DNodes (Data Nodes) provide physical storage capacity and data persistence. The two tiers are interconnected via NVMe fabric.

The cluster used for the benchmarks has a 24x16 configuration, meaning 24 CNodes and 16 DNodes. According to CoreWeave, this design allows performance to scale independently by adding CNodes, and capacity by adding DNodes.

The company also points to GPU utilization data from its monitoring infrastructure. It reports sustained 100% core utilization, with only minor dips during initial data loading and checkpoint operations, which it presents as evidence that GPUs are not waiting for data.

How the tests were run

CoreWeave used Flexible I/O (fio), which it describes as an industry-standard benchmarking tool for storage systems. Tests ran in CoreWeave's default Ubuntu 22.04 environment and were orchestrated through Slurm across its hybrid Slurm/Kubernetes infrastructure.

• Read tests: simulate mixed sequential and random I/O across block sizes from 32M to 1k. The reported results are an average of sequential and random performance.
• Write tests: focus on sequential writes to replicate checkpoint operations, again across block sizes from 32M to 1k.

The benchmarks used CPU-driven I/O rather than GPU-direct storage access. They were run on NVIDIA H200 GPU compute nodes, each with eight GPUs, to keep the configuration consistent with production environments. CoreWeave says complete test configurations and execution scripts are available in its public repository so that others can reproduce the results.

What it means for AI teams

CoreWeave cautions that storage performance can rarely, if ever, be compared one to one across platforms, since access patterns, dataset size and underlying infrastructure all matter. Its stated aim is to let customers run their own benchmarks, evaluate storage performance and find areas to optimize.

The company argues that storage which keeps pace with GPUs can improve training efficiency and reduce costs tied to idle GPU time. The source text provided here is cut off before the detailed benchmark result tables, so figures beyond the one GiB/s per GPU headline are not covered in this report.