Aleph Alpha: Leaked Benchmark Data Inflated HumanEval Scores by 45 Points

What the scan found

Before mid-training Kolibri, Aleph Alpha scanned its whole mid-training pool of 8.3 billion documents against every benchmark in its evaluation suite. According to the company, it removed 1.5 million documents that quote a benchmark item nearly verbatim. Many of them contain benchmark training problems rather than test items.

To measure what those documents were worth, the team put them back into a proxy run. The result, according to Aleph Alpha, was an inflation of HumanEval by 45 points and MMLU by 28. The gain reached even questions that never leaked.

The reference set came from the company's evaluation framework task registry: 226 tasks and 6.1 million instances, including train, validation and test splits. It covers MMLU, MMLU-Pro, ARC, HellaSwag, TriviaQA, GSM8K, MATH, HumanEval, MBPP, SQuAD and German counterparts. 64 of the 226 tasks are German. A document matching any task is dropped, and a match report with the quoted evaluation text is stored beside the data so each removal can be reviewed.

Why contaminated scores mislead

Aleph Alpha argues that a benchmark is a small sample standing in for a much larger space of problems. The 1,319 questions of the GSM8K test set, for example, stand in for grade-school math in general. Once a model has seen that sample, the score can rise without any change in underlying capability.

The post cites outside evidence. Scale AI researchers wrote GSM1k, a set of problems matched to GSM8K in style and difficulty, and some models scored up to 8 percentage points lower on it than on the original (13 in the first version of the paper). LiveCodeBench found that DeepSeek-Instruct and GPT-4o did worse on LeetCode problems published after their release and cutoff dates. The post also notes that of 30 developers surveyed, only 9 report train-test overlap.

There is an internal reason too: Aleph Alpha builds data mixes by comparing benchmark scores, so an inflated score can bias that choice.

How the detector works

The company integrated the decon algorithm released by AI2 for OLMo 3 mid-training into its own data curation pipeline. Questions are indexed as 5-grams and answers as 3-grams, each weighted by inverse document frequency, so shared boilerplate such as a prompt template counts for zero. To keep a scan of trillions of tokens affordable, the detector looks up only every tenth 5-gram. According to the post, this cannot miss a full copy of a question of 14 tokens or more.

A matched span is scored by question overlap (weight 0.75) and by whether the answer appears nearby (weight 0.25). A document is flagged when any instance scores 0.8 or above. Instances under 50 tokens face a stricter bar, rising to a perfect match at 20 tokens, and instances shorter than 20 tokens are not indexed at all.

Aleph Alpha drops whole documents instead of cutting out the matched part, as GPT-3 did with 200 characters around each colliding 13-gram. The company says that approach leaves a hole where the evidence was, making results harder to audit.

The blind spots

The detector itself scored at most 60 percent of the leaked MMLU questions above its threshold, and 13 percent of the leaked TriviaQA questions. The rest were removed only because they sat in the same files as caught material.

Decontaminating mid-training was also not enough. Memorization from pre-training, whose data Aleph Alpha did not decontaminate, resurfaced through the decontaminated mid-training: the model still recited HumanEval solutions it had learned there. The company says it noticed this only because it kept a close eye on benchmark scores during the production run.

What it means

For anyone reading benchmark tables, the post is a reminder that a high score can reflect memorization, and that filtering one training stage may not be sufficient. Aleph Alpha also notes that its task count is already out of date, since each new benchmark added to the registry means the pool must be scanned again.