Aleph Alpha: Machine-Translated German AI Benchmarks Often Measure the Wrong Thing

What a benchmark score actually stands for

An LLM evaluation, also called an eval or a benchmark, is a collection of thousands of questions with known answers. The model answers them, and the percentage it gets right is its score. According to Aleph Alpha, common evals cover school science (ARC), everyday common sense (HellaSwag) and general knowledge (MMLU), and nearly every model release reports scores on such tests.

The authors argue that a score reflects two things: what a model knows and what it can do. Isolating a single capability is difficult. Their example is a GSM8K math problem about a woman eating a 3 egg omelet every morning. Solving it takes arithmetic, but also the knowledge that a week has seven days and a dozen is twelve. In the company's words, no eval measures a single capability in isolation, so every score reflects a mix. Translating an eval into German can change that mix, and with it what the score means.

Why German models need German evals

Aleph Alpha says it builds models for German-speaking users that work in German and in the context of German culture, law and society. An earlier post from the company argued that such models need German training data, and that translated English data is not enough. The researchers say the same applies to evals. Nearly all standard evals are in English, and the widely used German versions are machine translations.

The team illustrates the limits of translation with a riddle: what can you call your sweetheart and also spread on toast? In English the answer is "honey". The literal German translation, "Honig", is not a term of endearment, so the riddle fails and a different one would be needed to test the same skill.

Three levels of mistranslation

The most obvious problem is incorrect translation. Aleph Alpha groups errors into three levels of severity, based on how much of the original meaning survives.

• Wrong wording, same question. A single term is mistranslated. In an ARC item, "consumer" became "Verbraucher", an economic consumer, instead of the ecological term "Konsument". An MMLU item contained "prim-powriger Ordnung", which is not a German word. A TruthfulQA item rendered "spend" as "verbringen", meaning to spend time. The score then partly measures how well a model can reconstruct the English original.
• Clean wording, different question. The German is correct but asks something else. An MMLU question beginning "As of 2016" became "Ab welchem Prozentsatz", turning it into a threshold question. Nothing signals the change, so the model is scored against the answer to a question it was never asked.
• Broken sentence, no meaning. Word order or grammar fails entirely. One TruthfulQA item from Eurolingua repeated part of a sentence, which the authors attribute to the translation system stuttering. Such items add noise, meaning random variation in the score that says nothing about a model's capabilities.

The examples come from translated sets labelled LeoLM, Eurolingua, Wino-X and Global-MMLU.

When grammar alone answers the question

The second problem affects multiple-choice and fill-in-the-blank evals. English words barely change form, while German nouns have gender and articles and adjectives change to match. The beginning of a sentence therefore restricts how it can continue. In the English blank "I ate the ___ (hamburger / melon)", both options fit. In German, "Ich aß den ___", only "Hamburger" is grammatically correct, because "Melone" would need the feminine article "die".

Translating a question and its options separately breaks their agreement, either for all options or only some. The authors say question and options can drift apart. In ARC and HellaSwag examples, no option continues the German question grammatically. In two MMLU examples, from Global-MMLU and Eurolingua, only part of the item was translated. Either way, the model must first guess how the item was meant before answering. The source text provided here breaks off during these examples, and the third problem, questions rooted in American culture, is named but not detailed.

What it means for users

The findings suggest that a German benchmark score cannot be read as equivalent to its English counterpart without checking the translation. For developers and buyers comparing models on German tasks, a score may partly reflect a model's ability to decode flawed German or exploit grammatical clues. Aleph Alpha's conclusion, in its title, is that German benchmarks need more than translation.