DL0089 RLHF: Human Feedback Training

Reinforcement Learning from Human Feedback (RLHF) is a fine-tuning procedure that optimizes a language model against a learned model of human preferences instead of against a hand-written loss. It exists because the behaviors we actually want (helpful, honest, non-toxic, well-formatted answers) have no differentiable definition: next-token cross-entropy on scraped text rewards imitating the corpus, not satisfying a user. The standard recipe has three stages: supervised fine-tuning on demonstrations, training a reward model on human comparisons of pairs of responses, then optimizing the policy with an RL algorithm such as PPO against that reward, with a KL penalty pinning the policy near its starting point. Humans never grade individual tokens; they rank whole responses, and the reward model is the mechanism that turns those sparse rankings into a dense per-sequence training signal. InstructGPT is the canonical demonstration: labelers preferred outputs from the 1.3B InstructGPT model to those from the 175B pretrained GPT-3 base model despite roughly 100x fewer parameters, so the gain came from the objective rather than from scale.

(1) Three-Stage Recipe: SFT → reward model → RL fine-tuning, where each stage consumes a different kind of human data (demonstrations, then comparisons, then only prompts).
(2) Preferences Are Cheaper Than Demonstrations: asking “which of these two answers is better?” is faster and gives higher inter-annotator agreement than asking someone to write the ideal answer, and the Bradley-Terry model converts those binary comparisons into a scalar reward.
(3) The KL Anchor Is Load-Bearing: the reward being optimized is a learned proxy, so the objective subtracts times the KL divergence to the SFT reference policy; remove it and the policy walks off-distribution into text the reward model scores highly and humans hate.
(4) Reward Hacking Is The Default Failure: proxy reward keeps rising while true human preference peaks and then declines, which is why production pipelines collect fresh comparisons on the current policy’s own samples rather than training once on a frozen dataset.

Figure 1: The three stages and the data each one needs. Stage 3 requires no new labels, only prompts, because the reward model has absorbed the human judgments, and the SFT checkpoint serves double duty as initialization and as the KL reference.

The reward model is a copy of the transformer with the language-modeling head replaced by a scalar head, trained so that the preferred response scores higher than the rejected one. Because only differences of rewards appear in the loss, the reward scale and offset are unidentifiable, which is why RLHF implementations whiten or normalize rewards per batch before computing advantages. In stage 3 the policy samples completions for a prompt, the frozen reward model scores each one, and PPO takes a clipped policy-gradient step; the KL term is usually implemented as a per-token penalty folded into the reward rather than as a hard constraint. A naive PPO setup keeps four networks resident (policy, reference, reward model, value head), so memory and orchestration cost is the practical reason many teams reach for a direct preference method instead.

Mathematical Formulation:

Where:

• is the Bradley-Terry pairwise loss and is the logistic function, so the reward model is trained as a binary classifier over response pairs.
• is a prompt from the dataset , and and are the human-preferred and rejected responses to it.
• is the learned scalar reward model with parameters ; only reward differences are identified, so its absolute scale is arbitrary.
• is the policy being optimized and is the frozen SFT policy; sets how far the policy may drift, with typical values near to .
• is the KL-shaped reward actually fed to PPO, and is the on-policy objective, with the expectation taken over responses sampled from the current policy.

Figure 2: Illustrative overoptimization curve: the reward model score climbs without bound as the policy drifts, while measured human preference peaks and falls. and early stopping exist to keep training left of that peak.