Aleph Alpha details two-scale expert balancing behind its Kolibri 1 MoE model

A very sparse model that depends on balanced experts

Kolibri 1 is a sparse Mixture-of-Experts (MoE) language model with 78B total parameters and roughly 3.5B active parameters per token, which is 4.5% of the total, according to Aleph Alpha. In each MoE layer, a learned router picks 6 of 384 routed experts for every token.

The company says training a model this sparse stably at scale depends heavily on an effective expert load-balancing scheme. The router decides how much training data each expert sees and how evenly work is spread over devices. If it keeps favoring a few experts, the others receive too few tokens to learn well, and part of the model's capacity goes unused.

There is also a hardware side. Experts may run on different devices with expert parallelism, during both training and inference. If too many tokens go to one device, the others have to wait for it. Aleph Alpha shows data from a single pre-training run in which higher local imbalance is associated with lower training throughput, consistent with devices waiting for overloaded experts.

Two mechanisms for two scales

The recipe targets balance at two levels. Global balance covers all tokens in a full optimizer step, so every expert gets enough gradient signal. Local balance covers each microbatch, to avoid bottlenecks in expert-parallel execution.

• Exact Quantile Balancing (EQB): extends Quantile Balancing (QB) and sets per-expert routing biases from exact quantiles of the router logits over the global batch.
• Load-Error Injection (LEI): injects each expert's load error in the local microbatch directly into the router's gradients.

Kolibri pre-training uses EQB together with bounded LEI. Aleph Alpha says LEI is a gradient-based correction closely related to the GShard auxiliary loss.

To measure imbalance, the post uses maximal violation (MaxVio), the largest relative overload among experts. A value of 0 means perfect balance, and 0.5 means the busiest expert gets 1.5 times its even share.

Why quantiles instead of fixed-step updates

Load-balancing methods generally fall into two families. Auxiliary losses, such as the GShard loss, add a balancing term to the training objective. Bias adjustment sets per-expert routing biases outside gradient descent. This includes DeepSeek's auxiliary-loss-free controller, which moves each bias by a fixed step, and QB, proposed by Jianlin Su and used in MARIN and Kimi K3.

Aleph Alpha explains the problem with top-K routing: experts compete for each token's K slots, so raising one expert's bias can displace assignments from others. DeepSeek's controller updates each bias from that expert's own load error and does not explicitly account for displaced assignments. A small update rate limits these side effects, but loads approach balance only gradually, over many steps.

QB instead needs no update rate, because it sets each expert's bias directly from the logit quantile matching its target load. The post rewrites top-K as a threshold rule, which lets each expert's load be adjusted independently once per-token biases are held fixed. EQB computes these global quantiles exactly across ranks.

What it means

The post is a description of a training recipe rather than a product release. It points to a full paper for derivations and reports ablations on a 7.5B-parameter proxy model trained for 100B tokens. For teams building highly sparse MoE models, it documents how one lab handles both data starvation of individual experts and device-level stalls in expert-parallel setups.