Apple Study Questions Whether LLM Judges Reflect Human Writing Preferences
Apple researchers have examined how reliably large language models can judge whether generated writing matches a user’s preferred style. The study finds that judge performance varies by task and evaluation criteria, while general model ability is only weakly linked to judging quality.
- Apple researchers evaluated LLM judges for personalized writing tasks.
- Judge quality had a weak correlation with MMLU ability, with Pearson’s ρ ≤ 0.2.
- Performance varied by writing task and evaluation criteria.
- Paired comparisons were more consistent and performant than independent scoring.
- The strongest LLMs did not most reliably reproduce human evaluations.
Testing judges on personalized writing
The paper, titled Evaluating the Evaluators: Investigating LLM Judges for Personalized Writing Style Assessment, was published in November 2026 and accepted at the Can We Trust AI Evaluation? Workshop at NeurIPS 2026. The authors are Natalie Mackraz, Andrew Silva, Aini Putkonen, Brihi Joshi, Joelle Alcaidinho, Arnav Arora, Barry-John Theobald and Katherine Metcalf.
The research focuses on cases where a user’s preferred writing style is known, such as an “upbeat” tone, and an LLM judge is asked to assess whether generated text follows that preference. Apple describes this as an important problem for personalized writing tasks, where a system must evaluate more than general quality.
To investigate the issue, the researchers generated text with four LLMs across three long-form writing tasks. They then collected human labels identifying which of two responses was better, gathered pairwise-ranking annotations from judge LLMs and compared the human and model evaluations.
Model strength was a weak predictor
The study reports that judge quality correlated weakly with general LLM ability as measured by MMLU, with a Pearson’s ρ of ≤ 0.2. Performance also varied according to both the writing task and the evaluation criteria, including the difference between general helpfulness and style personalization.
These findings suggest that a model’s broad reasoning or commonsense ability does not by itself show how well it will evaluate personalized writing. An evaluator may perform differently depending on what it is judging and which standard it is asked to apply.
Apple’s researchers also found that LLM evaluators were more consistent and performed better when comparing two responses directly than when scoring responses independently. The result supports pairwise comparison as a useful method for evaluating generated writing in the study’s setting.
Why evaluation design matters
The paper also challenges the assumption that the “strongest” LLM will produce the most human-aligned judgments. The researchers found that the strongest models did not most reliably reproduce human evaluations. They hypothesize that this may relate to exaggerated sensitivity to subtle details that human annotators and weaker LLMs ignore.
For developers building writing systems around user preferences, the findings point to the need to test the evaluator for the specific writing task and objective rather than selecting a judge based only on general model capability. The research also indicates that the choice between independent scoring and paired comparisons can affect consistency and performance.
The study does not present a single universally reliable LLM judge. Instead, it shows that assessing personalized writing requires comparing model judgments with human labels and examining how the evaluation setup changes results.
