Theoretical Limits of Language Model Alignment

What happened

A team of researchers established the theoretical upper bound for how much language model alignment can improve reward, given a fixed KL-divergence constraint. They derived a closed-form formula for this limit and created a practical estimator to predict alignment gains using only samples from the base model. Their analysis compared several alignment methods, finding that best-of-NN alignment nearly achieves the theoretical maximum, while PPO and GRPO methods fall short. The study also demonstrated that using reward ensembling can reduce the risk of reward hacking.

Key details

• The theoretical limit is defined by a Jeffreys divergence term, not the previously assumed square root of KL-divergence.
• Best-of-NN alignment closely approaches the maximum achievable reward improvement under the constraint.
• PPO and GRPO alignment methods are substantially less effective than best-of-NN in reaching the theoretical limit.
• A practical estimator was developed that predicts alignment gains from base model samples.
• Reward ensembling was proven to mitigate reward hacking, especially when the KL penalty is low or reward errors are high.

Why it matters

Developers of language models and researchers working on alignment now have a precise benchmark for the maximum reward improvement possible under KL-divergence constraints. This allows them to assess how close their alignment methods come to the theoretical optimum and to identify when further algorithmic improvements are needed. The findings also provide a concrete method to estimate alignment gains and a validated approach to reduce reward hacking, affecting the design and evaluation of safer, more reliable language models.