Apple study: LLMs and humans both struggle with translation quality annotation
Apple researchers report that large language models can sometimes agree with human annotators more than humans agree with each other when judging machine translation quality, but that both remain unreliable in most settings. The paper, from Hala Almaghout, Christian Federmann and Qin Gao, appears at EMNLP.
- Apple researchers compared LLM and human annotators on MQM and ESA translation quality schemes.
- Tests used a long-context set of 70 language pairs plus public WMT23 and WMT25 data.
- LLM-human agreement exceeds human-human agreement for some tasks, but both vary widely and remain unreliable for most settings.
- Humans struggle with fine-grained MQM and low-resource languages; LLMs struggle with minor errors, wrong language variants and error spans.
- The authors point to human-LLM collaborative annotation pipelines as a possible improvement.
What the researchers tested
LLMs are widely seen as a more efficient and cost-effective alternative to human judgment for evaluating machine translation (MT), according to the paper. MT evaluation covers many language pairs, domains and levels of annotation granularity, so the authors argue that LLMs need to be thoroughly checked across all of these dimensions before they can reliably replace human evaluation.
The team examined two prominent MT quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA). For each, they compared how closely LLM annotations agreed with those of human annotators. They looked at both score agreement and error span annotation agreement.
The data included a long-context test set covering 70 language pairs, as well as the publicly available WMT23 and WMT25 data.
Mixed results for models and humans
According to the paper, LLM agreement with human annotators exceeds the agreement between human annotators for some evaluation tasks. However, the results vary substantially across annotation schemes, language pairs and domains, and the authors say they remain unreliable for most settings.
The study also identifies where each group of annotators runs into trouble:
- Humans are particularly challenged by fine-grained MQM annotations and by low-resource language pairs.
- LLMs struggle with minor errors, wrong language variants and error span annotation.
The finding that humans disagree with each other is notable because human judgments are often treated as the reference point against which automated methods are measured.
What it means for MT evaluation
The authors say their results point to room for improvement in both human and LLM annotation. They suggest that human-LLM collaborative annotation pipelines could address the reliability issues the paper identifies, though they frame this as a possibility rather than a tested solution.
For teams that rely on LLMs to score translations, the study is a caution against assuming consistent quality across all language pairs, domains and annotation types. The paper does not show LLMs as a drop-in replacement for human evaluation, and it also shows that human annotation is not uniformly dependable, especially for fine-grained error labeling and low-resource languages.
