DL0176 Online vs Offline DPO

Both variants minimize the same Bradley-Terry log-loss over the implicit reward , so the difference is not the objective but the distribution the training pairs are drawn from. Offline DPO consumes a dataset collected once from some behavior policy, which makes it a purely supervised procedure: four log-probability forward passes per pair, no decoding, and reference log-probs that can be cached before training starts. Online DPO decodes two fresh completions from the current policy at every step, has them ranked by a preference oracle (reward model, LLM judge, or human), and applies the DPO gradient to that on-policy pair, so the data distribution moves as the model moves. The generalization gap follows directly from this. DPO’s derivation only pins the implicit reward down where the data has support, and a frozen dataset’s support stops moving while the policy keeps drifting, so on unfamiliar prompts the model is being shaped by an extrapolated reward that no label ever corrected. On-policy sampling forces the coverage mismatch back toward 1 by construction, so the KL-regularized objective is enforced exactly at the responses the model will actually emit.

(1) Same Loss, Different Sampler: the gradient formula is identical, but offline draws from a fixed while online draws them from and labels them on the fly.
(2) An Oracle Becomes Mandatory: online DPO needs a ranker in the loop, so the quality ceiling shifts from the dataset to the reward model or judge.
(3) Coverage Versus Exploration: offline guarantees scale with a concentrability coefficient over the whole response space, while on-policy data only needs local coverage around the current policy.
(4) Off-Policy Drift Is Cumulative: every gradient step moves further from , so the last epoch of offline training is the most off-policy and the least trustworthy.
(5) Cost Is Decode-Bound: an online step adds two autoregressive generations plus an oracle call, typically 5x to 10x the wall-clock of an offline step at the same batch size.
(6) Hybrids Dominate In Practice: iterative or batched online DPO regenerates the preference set every few thousand steps, recovering most of the on-policy benefit at a fraction of the sampling overhead.

Mechanically, offline DPO is a four-forward-pass classification problem. You score the chosen and rejected completions under the policy and under the frozen reference, take the difference of differences, and push it through a log-sigmoid. Nothing in that loop ever asks what the model would say today. Online DPO inserts two extra stages before the loss, decode → rank → update, and the decode stage is what costs money: generating two 512-token completions is roughly a thousand sequential memory-bandwidth-bound forward steps, whereas the loss itself is four parallel prefills. The payoff is that the pair being contrasted is a pair the model genuinely produced, so the gradient always removes probability mass from an error the model is currently making rather than from an error some other model made months ago.

Figure 1: The structural difference is one arrow. Offline DPO is an open chain whose data never changes, so the mismatch between the dataset and the policy grows monotonically; online DPO closes the loop, which is what keeps the preference signal on-policy at the price of two decodes and one oracle call per prompt.

Mathematical Formulation:

Where:

• is the implicit reward that DPO optimizes in place of an explicit reward model, and is the inverse KL penalty controlling how far may move from .
• is the prompt, the preferred completion, the rejected one, and their implicit-reward margin.
• is the logistic function, so is the per-pair gradient weight: pairs the model already ranks correctly contribute almost nothing.
• is the contrastive direction.
• is the fixed behavior policy that produced the offline corpus, typically an SFT checkpoint or a different and often stronger model.

Coverage Requirement:

The offline sample complexity carries a factor of , the worst-case density ratio between the target policy and the data-collecting policy. On out-of-distribution prompts, assigns near-zero mass to the region the trained policy now occupies, so explodes and the bound says nothing. This is visible empirically as likelihood displacement: DPO frequently drives down the log-probability of the chosen response as well as the rejected one, and the displaced mass lands on unlabeled completions that the loss never inspects. On-policy sampling closes exactly that hole, and the DeepMind analysis of the online-offline gap points the same way: offline algorithms remain excellent at classifying preferences while degrading at generating preferred text, and scaling offline data does not close the gap.

Figure 2: Offline training grades the policy where the data is, not where the policy is. Once has drifted, most of its probability mass lies in the shaded region with no labeled pairs, and the implicit reward there is pure extrapolation. Online sampling drags the labeled region along with the policy, which is why the coverage ratio stays near 1 even as the model changes.