Aleph Alpha details how architecture choices drive LLM cost, with Kolibri as reference

Three quantities that set cost

According to Aleph Alpha, the effect of an architectural choice on quality has to be measured empirically, but the costs follow directly from the configuration. The post focuses on three quantities. Total parameters set the weight-memory footprint, and with it the minimum hardware needed to deploy a model. FLOPs per token, multiplied by the training-token horizon, set training compute, and also serve as a proxy for prefill cost, which is typically compute-bound. Sequence-mixer state, such as the key-value (KV) cache in an attention-based model, determines how many sequences and how much context fit in a decode batch, and how many bytes each decode step must read.

Which of these becomes the serving bottleneck depends on the workload, the authors write. Small-batch, short-context decoding is limited by reading the weights. Prefill and large-batch decoding of short queries are limited by compute. Long-context decoding is limited by reading the sequence-mixer state.

Parameter allocation in MoE models

The post derives each quantity for Mixture of Experts (MoE) decoders and compares recent open-weight models. It looks at how active parameters are split between the feed-forward network (FFN), the sequence mixer and the language modelling head, how FLOPs per token are allocated and scale with sequence length, and how much state each sequence holds.

Aleph Alpha notes that in MoE architectures total and active parameters can differ by an order of magnitude. Total parameters mainly affect memory footprint, while active parameters typically account for most of the FLOPs during training and prefill. At small batch sizes they also largely determine how many weight bytes each decode step reads.

The derivations are concrete. A SwiGLU expert has three linear projections (gate, up and down), giving 3 × d × d_e parameters per expert. Active routed parameters are counted by replacing the number of experts with the top-k experts actually used. Multiple shared experts are equivalent to one shared expert of larger width, since they are always active. The router, which maps the residual stream to one logit per expert, is described as a very small part of total parameters. For attention, the post gives the projection parameters as 2 × L_attn × d × (d_q + d_kv), with multi-head attention and multi-query attention as the two edge cases of grouped-query attention. It also covers normalisation layers and the embedding and LM head.

Kolibri compared with other open-weight models

The interactive explorer is described as a simplified subset of an internal tool Aleph Alpha uses when designing architectures, meant to quantify trade-offs before committing to a configuration. It is inspired by Sebastian Raschka's LLM Architecture Gallery and starts with two Aleph Alpha models, Kolibri Origin and Kolibri. The company says Kolibri was recently released under the Apache 2.0 licence. Users can adjust sliders or add their own configuration.

A table in the post lists total and active parameters per token. Kolibri has 78.1B total and 3.5B active parameters, or 4.4% active. Kolibri Origin has 30.6B total and 3.3B active, or 10.7%. Other entries include Nemotron 3 Nano (31.6B total, 3.2B active), GPT-OSS 120B (116.8B total, 5.1B active) and GPT-OSS 20B (20.9B total, 3.6B active). At the large end are DeepSeek V4-Pro with 1.57T total and 48.8B active parameters, and Kimi K3 with 2.78T total and 104.2B active.

The table counts every stored weight of the text decoder as total. Active parameters exclude the input embedding, because it is a lookup rather than a matrix multiplication.

What it means for model builders

The post is aimed at people designing or evaluating language models. It shows that deployment hardware needs, training compute and long-context serving limits can be estimated from a configuration before any training run. The comparison table also shows how widely the share of active parameters varies among current open-weight models, from 2.9% for DeepSeek V4.1-Flash to 17.3% for GPT-OSS 20B. Quality effects, the authors stress, still require empirical measurement.