Aleph Alpha Scales 30B-A3B MoE Pre-Training to 512 B200 GPUs at 35.3% MFU

A hierarchical search instead of brute force

The post, written by Jordan Sassoon and dated 30 September 2026, starts from a simple point: adding more GPUs does not automatically make training faster. Aleph Alpha says efficiency degrades quickly if the software is not carefully tuned. A naive grid search over the many efficiency-related hyperparameters is, in the company's words, prohibitively expensive at scale.

Its answer is a hierarchical approach. The team shrinks the search space with small-scale experiments before increasing the GPU count. The company notes that final training configurations are sometimes disclosed by labs, but the reasoning that leads to them rarely is, and says it wants to show that process.

Training runs on Aleph Alpha's own optimised fork of torchtitan. The target cluster consists of 64 nodes connected via InfiniBand, with 8 GPUs per node connected via NVLink. The post says the system- and kernel-level optimisations that make 35.3% MFU possible are mostly out of scope, as it focuses on the scaling methodology itself.

Measuring efficiency: TPS and MFU

Aleph Alpha uses three metrics: tokens per second per GPU (TPS), model FLOPS utilisation (MFU) and GPU memory usage. TPS is the number of tokens processed by a single GPU divided by wall time. The company calls it a crude but honest measure of training speed and the one it optimises against.

MFU expresses the same measurement as a fraction of the hardware's theoretical peak floating-point throughput. The post cautions that MFU comparisons between models can be murky, for example because mixed-precision training changes what the peak FLOPS are. Aleph Alpha trains in BF16, apart from FP32 gradients, so all MFU figures use BF16 peak FLOPS as the denominator. It still reports MFU because it is the industry standard.

Memory usage is measured as peak reserved memory divided by total device memory. The team uses PyTorch's reserved memory rather than allocated memory because reserved memory dictates hard out-of-memory failures.

Three levers to stay compute-bound

The aim is to be compute-bound, meaning GPUs are limited by how fast they can do mathematical operations rather than by how fast data moves between them. Communication does not directly advance the model's computation, so idle GPUs waiting on data waste capacity.

The post names three degrees of freedom for tilting the communication-to-computation ratio:

• the parallelism scheme
• the activation checkpointing technique
• the local batch size

On parallelism, the company draws on Hugging Face's Ultra-Scale Playbook and focuses on two dimensions. Fully Sharded Data Parallelism (FSDP, or ZeRO-3) splits weights, gradients and optimiser states across the GPUs of a group, which together form one full model replica. Data Parallelism (DP) refers to the number of model replicas, equal to the number of independent FSDP groups. The excerpt describes the all_gather and reduce_scatter operations used within FSDP, and the all_reduce operation introduced by DP.

Why it matters beyond large clusters

Aleph Alpha argues the lessons are not limited to labs with hundreds of top-end GPUs, because many of the tools and ideas are scale-agnostic. Higher efficiency saves money and, as the post puts it, cuts down time.

The remaining sections, as outlined in the post, cover using 16-GPU experiments to cut down the hyperparameter space, profiling to see which operations the model performs and whether they are efficient, and the final scale-up from 16 to 512 GPUs, where the company says scaling cost is nearly free. The available text ends before those results are presented in detail, so the headline figures of 35.3% MFU and 6% below linear scaling are the company's own claims.