Modal Clusters reach general availability with one-line multi-node GPU access

What Modal Clusters offer

Modal says it has spent the past 1.5 years battle-testing Clusters, which it describes as a new primitive: a group of containers treated as one unit of work. Users request a cluster by adding @modal.clustered to a function, specifying the number of nodes and whether to enable RDMA. In the example in the announcement, a function requests size=4 with rdma=True and eight B300 GPUs per node.

Clusters work with existing Modal primitives. Checkpoints can be written to Volumes, data can be loaded through Cloud Bucket Mounts, and jobs can be orchestrated with Queues. Beyond that, Modal lists three features:

• Communication between nodes via InfiniBand verbs at up to 6.4 Tbps, automatically configured for PyTorch and NCCL
• Code running on a cluster within seconds of requesting it, billed by the second
• Automated GPU health assurance and remediation, now extended to RDMA health

Modal contrasts this with on-demand cluster access elsewhere, which it says means paying by the hour or taking a reservation. On Modal, users run whenever they want and pay only for what they use, according to the company.

A new gang scheduler and RDMA support

Modal says its traditional scheduler is greedy: nodes poll for work and decide per node whether to pick up an item. That approach does not suit multi-node jobs, where the unit of scheduling grows from one node to N nodes. The company therefore built a new scheduler with an explicit observe, plan and act loop that looks at the whole fleet and decides on a placement for every pending cluster. Modal says the rest of its scheduling systems are moving in the same direction.

On each iteration, the gang scheduler collects metadata for all multi-node workloads, assigns them to nodes grouped by availability zone and network ID, spins up new capacity for workloads that do not fit, and sends schedule notifications to live nodes. Because it draws on the same capacity pool as the rest of Modal, clusters can be acquired in seconds, which the company calls the fastest time to get a multi-node cluster on the market.

Modal argues that RDMA matters because multi-node workloads move large amounts of data. It says each training step of GLM 4.7 must sync about 717 GB of BF16 weights from the trainer to the rollout engine, taking nearly two minutes over 50 Gbps TCP but under two seconds over RDMA. For inference, it says prefill-decode disaggregation on Llama 3.1 70B moves about 10 GB of KV cache per 32k-token prompt, which must arrive within a time-to-first-token budget of a few hundred milliseconds.

RDMA is hard to support because each cloud provider implements it differently, with different drivers, environment variables and userspace libraries. Modal says it hides this behind the single flag rdma=True and that any workload using PyTorch with NCCL works out of the box. It places all nodes of a cluster on one RDMA fabric in one zone.

Modal runs a multi-tenant environment and prefers the gVisor container runtime over runc. Since gVisor lacked RDMA support, Modal says it built it and upstreamed the changes so others in the open-source ecosystem get access to secure RDMA.

Customers and what it means

Modal cites several customers. Decagon, which builds AI agents for customer experience, fine-tuned open models with up to a trillion parameters using Miles on Modal Clusters. Modal says it contributed LoRA support to Miles and helped establish a stable, open training recipe. Decagon research lead Cyrus Asgari said Modal Clusters "helped us train at that scale, so we could focus on improving the quality and reliability of our agents."

1x pre-trains the world model behind its home robot NEO on multi-node B300 clusters with RDMA, and runs fleets of single-node jobs for evaluations. Modal says the same scheduler handles both, so 1x can burst to hundreds of GPUs and scale back down. Runway uses Clusters to spread inference for its video models Gen-4.5 and Aleph 2.0 across multiple nodes, according to the announcement.

For teams that train or serve large models, the release offers a way to get multi-node, RDMA-connected GPU capacity without reservations or hourly commitments, with per-second billing, as Modal describes it. The claims on speed and pricing come from the company itself.