Aleph Alpha: Bits-per-Byte Metric Penalizes Short Answers, PriorBPB Fixes It

Why pre-training evaluation matters

Before a large language model can chat or follow instructions, it is pre-trained to predict the next piece of text across trillions of words. According to Aleph Alpha, this stage can make up most of a model developer's compute budget. The company says pre-training its recent Kolibri model took about 392,000 GPU hours on 768 NVIDIA B200 GPUs, roughly an order of magnitude more than the compute of its final reinforcement learning run.

Because of that cost, teams run many smaller experiments first, for example to decide which data to train on or which architecture to pick. Each experiment needs a score showing whether a change helped. Base models usually cannot hold a conversation or carry out multi-step tasks, since those abilities mostly come from post-training. Accuracy on such tasks can therefore be flat or noisy, and it depends heavily on prompt or answer formatting. A biased score, the post argues, can lead a team to pick the wrong recipe.

How BPB penalizes short answers

BPB gives the model a prompt and scores it on a fixed gold answer string one token at a time. The surprisal in bits is summed over the answer and divided by the answer's UTF-8 byte length, which keeps scores comparable across models with different tokenisers. Lower is better.

The author, Tom Burns, shows that this division creates a length effect. For the question "What is the capital of France?", Qwen3-4B scores 1.85 on "It is Paris.", 1.10 on "It is the city of Paris." and 0.77 on "It is Paris, the capital of France." OLMo-3-7B scores 1.38, 1.34 and 1.22 on the same variants. The meaning is the same, yet longer answers score better.

The post gives a reason. The longest Qwen3-4B answer costs only about 15% more bits than "It is Paris." (27.7 vs 24 bits), but it is almost 3x as long (36 vs 13 bytes). Within the MMLU full text task scored with Qwen3-4B, the group with the shortest answers (6 bytes on average) scores about 11x worse than the group with the longest answers (96 bytes on average).

The researchers fit each item's bits as a fixed overhead plus a per-byte rate. Corpus BPB then equals the overhead divided by the task's mean gold length, plus the rate. The fixed cost, largely driven by the surprisal of the first answer token according to the post, shrinks as answers get longer. Across the 19 tasks tested on multiple models, this term correlates strongly with mean gold length, while the per-byte rate barely does.

The authors also tested whether long-answer items are simply easier. Within a single task, the shortest fifth of items scored worse on every task tested, typically about twice as badly (median top/bottom quintile ratio of about 0.43). They add that mean-of-ratios aggregation adds further task-dependent inflation, enough on Qwen3 models to create an apparent but misleading inverse-scaling trend between the 0.6B, 1.7B and 4B models.

What PriorBPB changes

For each model and task, PriorBPB estimates a marginal per-byte cost curve from per-token surprisal, then averages that curve under one length prior shared across tasks. With each task's own length distribution as the prior, it equals corpus BPB.

Across more than 60 checkpoints from five model families, Aleph Alpha says PriorBPB is more length-neutral than standard BPB metrics. The dependence on gold answer length largely disappears, both when ordering tasks against each other and when comparing short and long answers within a task. The company also reports that PriorBPB correlates with accuracy more closely than standard BPB metrics, somewhat on base checkpoints and substantially on instruct checkpoints.

The affine relationship between total surprisal and answer length was documented earlier by Oostermeijer (2026), according to the post. PriorBPB is included in Aleph Alpha's open-source evaluation package, eval-framework.

What it means for model builders

Based on the post, comparisons of tasks using standard BPB largely reflect how long the gold answers are, so a low score on one task versus another may say little about relative difficulty. The post gives an example: Qwen3-0.6B scores an order of magnitude better on MMLU full text than on SQuAD2, which may reflect the tasks' gold-length distributions more than their difficulty.

The authors also note that the share of cost that scales with answer length varies from task to task, so no simple normalisation fits all of them. For teams choosing data mixes or architectures from small-scale experiments, PriorBPB offers a metric that is less tied to answer length and tracks accuracy more closely.