DL0196 Reward Model Overfitting

A reward model (RM) is a learned proxy for human judgement fitted on a finite set of preference pairs, so it fails in two distinct ways that need different defenses. The first is ordinary statistical overfitting: held-out pairwise accuracy typically plateaus in the 65-75% band and then degrades, which is why InstructGPT-era practice is a single epoch at a small learning rate, since a second epoch mostly memorizes annotator noise. The second and more damaging failure is over-optimization, where held-out accuracy still looks acceptable but the policy drifts into a region the RM never saw labeled, so the proxy score climbs while true human preference falls. The defenses are therefore spread across the whole loop rather than concentrated in the RM training script. Diverse prompts and on-policy responses widen the region where the RM is actually supervised, an ensemble of independently seeded RMs converts the unsupervised region into a measurable disagreement signal, and a pessimistic aggregate such as mean minus times the spread turns that signal into a reward the policy cannot profit from. A KL penalty to the frozen reference policy plus periodic relabeling of fresh on-policy samples keeps the optimizer inside the RM’s zone of competence.

(1) Two Different Failures: Classical overfitting is measured on held-out preference pairs, while over-optimization is a Goodhart effect visible only when the RM is used as an RL objective. Fixing one does not fix the other.
(2) Train Short, Regularize Hard: One epoch, learning rates around or below, early stopping on held-out pairwise accuracy, plus label smoothing or a margin term and an auxiliary language-modeling loss to keep the backbone from collapsing.
(3) Data Diversity Sets The Support: Prompts from real traffic, red-teaming, and synthetic edge cases, with responses sampled from several policy checkpoints and temperatures, so the RM is supervised where the policy will actually go.
(4) Multiple Annotators Bound The Ceiling: Human agreement of roughly 70-80% on subtle pairs caps achievable accuracy, so a model chasing 90% training accuracy is fitting noise, not preference.
(5) Ensembles Estimate Epistemic Uncertainty: to members whose pretraining seeds differ disagree most off-distribution, which is exactly where reward hacking begins.
(6) Pessimism Turns Uncertainty Into Reward: Optimizing a lower confidence bound or the worst-of- score delays the over-optimization peak instead of merely detecting it.
(7) Close The Loop: Iterative rounds that relabel on-policy samples attack the root cause, distribution shift, rather than its symptom.

Figure 1: Where each defense sits in the loop. Diversity widens the labeled support, independently seeded reward models turn unlabeled regions into a measurable spread , and the conservative reward makes the policy pay for that spread. The dashed feedback edge is the only mechanism that actually removes distribution shift rather than penalizing it.

Data diversity is the highest-leverage lever because it changes the support of the training distribution rather than the fit on a fixed support. Preference sets collected from one prompt source and one policy checkpoint produce an RM that is accurate on that slice and arbitrary elsewhere, and the policy will find “elsewhere” within a few thousand PPO steps. Two diversity axes matter most: the prompt distribution, which should mix production traffic, adversarial red-team prompts, and long-tail synthetic cases, and the response distribution, which should include samples from several checkpoints, several temperatures, and deliberately balanced lengths so the RM does not learn that longer equals better. Multi-annotator labeling on a subset gives the agreement rate, and that rate is the practical accuracy ceiling: driving training accuracy far past it is a direct measurement of memorization.

Mathematical Formulation:

Where:

• is the score margin between the preferred response and the rejected response for prompt , and is the logistic function, so is the Bradley-Terry negative log-likelihood.
• are the RM parameters: a pretrained or SFT backbone with a scalar head, usually trained for one epoch so never approaches zero.
• is the score of ensemble member , and their mean, which is also a mild variance reduction on the reward itself.
• is the ensemble spread, used as an epistemic uncertainty proxy: it stays small on the labeled support and grows on responses no annotator ever judged.
• is the pessimism coefficient, commonly in the range to ; recovers plain mean aggregation.
• is the KL coefficient (typically to ) and the divergence from the frozen reference policy, which bounds how far the policy may travel from the RM’s support.

Figure 2: Left: held-out pairwise accuracy peaks near one epoch while training accuracy keeps climbing, which is why single-epoch training with early stopping is the default. Right: the same RM used as an RL objective. The proxy score rises monotonically while the gold reward peaks and then declines, and pessimistic aggregation pushes that peak further out. Curves are schematic and follow the functional form fitted by Gao et al.

The right panel is the empirically important one, and its shape is well described by the over-optimization scaling law, where the gold reward is a function of the drift rather than of training steps.

Over-Optimization Law:

Here and are fitted constants that improve with RM size and with the amount of preference data, so a larger RM trained on more diverse comparisons does not merely score better, it peaks later. Ensembling contributes on two fronts. Averaging members reduces variance in the reward itself, which stabilizes advantage estimates, and the spread provides the only cheap in-loop signal that the policy has left the supervised region. In practice the disagreement metric is worth logging even if you never subtract it: a sudden rise in mean over sampled rollouts is an early warning that arrives before the gold reward turns over. Subtracting it, as in conservative or worst-of- optimization, costs a little early progress because uncertain but genuinely good responses are also penalized, and buys a substantially later peak. The critical implementation detail is that members must be diverse in their errors: fine-tuning five heads from one pretrained checkpoint with different shuffles yields correlated mistakes, so the spread underestimates true uncertainty precisely on hacked outputs.