CoreWeave posts 441,740 tokens per second on DeepSeek-R1 in MLPerf Endpoints

What CoreWeave measured

According to CoreWeave, it ran DeepSeek-R1 on a NVIDIA GB200 NVL72 system and sustained 441,740 output tokens per second of system throughput at 16,384 concurrent requests. The per-GPU figure of up to 6,496 tokens per second is, per the company's footnote, an unverified metric derived by dividing system throughput by the GPU count of 68. The system-level result was verified by MLCommons Association.

CoreWeave argues that throughput per GPU is the metric that most directly reflects the economics of a real inference service, because it maps to tokens served per accelerator and cost per GPU. The footnotes also list a time to first token (P95) of 5.77 seconds for the result.

The reference configuration for this workload targets a full 18-node, 72-GPU rack. CoreWeave submitted results on 17 nodes, using 68 Blackwell GPUs in a GB200 NVL72 rack. The company states that it posted the leading per-GPU DeepSeek-R1 throughput among NVIDIA GB200 NVL72 submissions.

About the benchmark and the model

Endpoints is described as MLCommons' newest benchmark. It aims to measure what earlier training and inference suites do not: how an inference service behaves under realistic, concurrent, online load. It reports the full throughput-versus-latency curve at scale, up to 16,384 concurrent requests, instead of a single data point.

The suite includes DeepSeek-R1, a 671-billion-parameter mixture-of-experts reasoning model that emits thousands of tokens per response. CoreWeave calls it the hardest serving workload in the suite. Its own runs averaged well over 2,000 tokens per request.

For reasoning models, CoreWeave says two metrics decide both user experience and delivery cost: throughput per GPU and time per output token (TPOT). TPOT is how fast a response streams once it starts, and for long outputs it dominates end-to-end latency. The company says that under maximum concurrency it held one of the lowest TPOTs in the round while carrying more load per GPU.

Production infrastructure, not a lab setup

CoreWeave stresses that the results did not come from an isolated environment tuned for one clean number. It says it followed NVIDIA's reference designs and used a standard production cluster, with the same network fabric, scheduler, storage and CoreWeave Mission Control orchestration that customers use. No separate benchmark cluster was created, according to the company.

The company lists the stack layers involved:

• Bare Metal servers give direct GPU access without a hypervisor layer.
• Networking is built on NVIDIA Quantum-2 InfiniBand.
• CoreWeave Kubernetes Service (CKS) is the base runtime, integrated with Mission Control.
• SUNK provides topology-aware scheduling to reduce cross-node communication.
• Distributed File Storage (DFS) serves DeepSeek-R1 weight loading across all 68 GPUs.

CoreWeave says this stack also runs its AI Inference products: Serverless Inference for pay-per-token access to a curated open-source model catalog, Dedicated Inference for custom weights with explicit GPU and runtime control, and Inference on CKS for fully self-managed serving.

What it means for customers

CoreWeave frames the result as guidance for capacity planning: sustained steady-state performance rather than a best-case spike, and the possibility of reaching the same or higher token output with a smaller GPU footprint. It also says higher per-GPU throughput at peak concurrency means more tokens served per accelerator-hour as traffic grows, and less need for extra GPU headroom to cover dropped requests.

Because the three inference paths share one foundation, the company says teams can expect the same predictable performance across them and can move between paths as workloads grow without replatforming. These are CoreWeave's own claims, and the per-GPU number is not a verified MLPerf metric.