Apple Paper Defines Theoretical Limits for Language Model Alignment

Apple researchers have published a NeurIPS paper defining the maximum reward improvement language models can achieve under a fixed KL-divergence budget. The work also finds that best-of-N alignment comes close to this theoretical limit, while PPO and GRPO remain substantially suboptimal in the researchers’ experiments.
- Apple researchers derive the maximum reward gain under a fixed KL-divergence budget.
- The proposed closed-form result uses a Jeffreys divergence term.
- A covariance-based estimator predicts gains from base model samples alone.
- Best-of-N closely approaches the theoretical limit in safety and summarization.
- PPO and GRPO remain substantially suboptimal in the paper’s experiments.
A bound for reward improvement
The paper, titled Theoretical Limits of Language Model Alignment, examines how alignment methods can improve a language model’s outputs to reflect human preferences while preserving the capabilities of the base model. Its authors are Lucas Monteiro Paes, Natalie Mackraz, Barry-John Theobald and Federico Danieli, with Paes and Mackraz making equal contributions.
Apple’s researchers focus on two widely used approaches. Reinforcement learning maximizes expected reward under a KL-divergence constraint, while best-of-N alignment selects the highest-reward output from N independent samples. The paper says that the fundamental limits of reward improvement under a KL budget have remained poorly understood.
To address that question, the researchers derive the maximum achievable expected reward gain for a fixed KL-divergence budget. Their first result is a closed-form expression for the optimal reward improvement. The expression is governed by a Jeffreys divergence term rather than the square root of KL divergence used in prior analyses.
Estimating gains and examining reward hacking
The researchers also reformulate the result as a covariance under the base model. According to the paper, this produces a practical estimator that can predict achievable alignment gains using samples from the base model alone.
The analysis extends to settings that use a proxy reward rather than an ideal reward. In this setting, the paper describes a gap between ideal and proxy alignment, known as reward hacking. The gap grows as reward error increases and as the KL penalty factor decreases.
The paper further proves that reward ensembling mitigates reward hacking. This provides a theoretical justification for a technique already used in practice, according to the Apple research page.
Implications for alignment methods
For its empirical analysis, the paper computes the KL-reward Pareto frontier for two language-model tasks, safety and summarization. Apple’s researchers report that best-of-N closely approaches the theoretical limit, while PPO and GRPO remain substantially below it.
Those results suggest that some commonly used alignment algorithms do not reach the best reward improvement available under the KL constraint. The researchers say their theoretical findings help explain several phenomena observed in alignment research and indicate that algorithmic improvements are needed to achieve optimal alignment without high inference costs.
For model developers, the work offers both a way to estimate potential alignment gains before running a full alignment procedure and a framework for comparing methods against a theoretical frontier. It also highlights a tradeoff between improving proxy-reward performance and controlling reward hacking when the KL penalty is reduced.
Apple: Theoretical Limits of Language Model Alignment


