CoreWeave and NVIDIA Train MLPerf GPT-3 Benchmark in Under 11 Minutes

The result and the hardware behind it

In a joint MLPerf Training submission, CoreWeave and NVIDIA ran the GPT-3 175B large language model benchmark on CoreWeave's publicly available supercomputing infrastructure. According to CoreWeave, the run used more than 3,500 NVIDIA H100 Tensor Core GPUs and finished in under 11 minutes. The company says the performance was more than 29x faster than the next best competitor, and that the run was also 4x larger in scale.

CoreWeave describes its supercomputer instances as one of the largest NVIDIA HGX clusters in the world. They combine HGX servers with NVIDIA H100 GPUs, Intel 4th Generation Xeon Scalable Processors, NVIDIA ConnectX-7 400Gb/s InfiniBand and BlueField-2 DPUs.

MLPerf Training 3.0 added GPT-3 175B, a large language model that powers services like OpenAI's ChatGPT and is based on the Transformer architecture. CoreWeave calls MLPerf the industry-standard benchmark for training and inference, and compares it to the "0 to 60" figure for cars. It says the benchmarks are peer-reviewed by AI leaders in academia, research labs and industry.

A production cluster built for Inflection AI

The cluster used for the submission is a production system built with NVIDIA Quantum-2 InfiniBand networking for Inflection AI. CoreWeave says Inflection AI donated the compute for the MLPerf tests. Inflection AI is the creator of Pi, a personal AI designed as a kind and supportive companion that offers conversations, friendly advice and concise information.

According to CoreWeave, Pi was trained on its NVIDIA H100 instances. Mustafa Suleyman, CEO and co-founder of Inflection AI, said: "Anyone can experience the power of a personal AI today based on our state-of-the-art large language model that was trained on CoreWeave's powerful network of H100 GPUs."

The cluster includes over 40,000 cables and 500 miles of InfiniBand fiber cables, CoreWeave says. It adds that the NVIDIA Quantum InfiniBand networking, used in all its H100 and A100 instances, lets GPUs communicate directly with each other with low latency at scale.

CoreWeave's argument about scale

CoreWeave says it was among the first providers to offer cloud instances with NVIDIA H100 GPUs, which became generally available to clients during the NVIDIA GTC event in March. It argues that hyperscalers were not built to provide this kind of compute on a contiguous scale, and that its specialized infrastructure offers bare-metal performance with matching storage, networking and software.

Brian Venturo, CTO and co-founder, put it this way: "Historically you didn't get more than 2,000-3,000 GPUs from a single cloud provider in a single location. CoreWeave is building installations that are 20,000+ NVIDIA GPUs in one location."

CEO and co-founder Mike Intrator said the results "prove that CoreWeave can deliver what we say we can deliver."

What it means for AI teams

The company presents the result as evidence that large-scale model training can run in a public cloud at speeds previously associated with dedicated systems. It says teams using CoreWeave Cloud can choose from a wider variety of NVIDIA GPUs and right-size their workloads, and that its compute is optimized for highly parallelized workloads.

Looking ahead, Intrator said much of the industry is focused on training today, but that models will soon need to be served as products. "We know that AI inference demand is about to explode," he said. The figures are CoreWeave's own account of the submission, and the source does not detail the competing results.