DL0108 PPO vs DPO vs GRPO

All three optimize the same underlying target, maximize a preference-derived reward while staying close to a frozen reference policy, and they differ in how many networks must be resident, how the advantage is estimated, and whether the training data is sampled from the current policy. PPO is the full actor-critic loop used in InstructGPT-style RLHF: it keeps four networks (trained policy, frozen reference, frozen reward model, trained critic), samples rollouts on-policy, and updates with a clipped importance ratio against token-level GAE advantages produced by the critic. DPO deletes the RL loop entirely by inverting the closed-form solution of the KL-constrained objective: the implied reward is , the partition function cancels inside a Bradley-Terry pairwise likelihood, and what remains is a supervised binary-classification loss on fixed pairs with only two networks and no sampling. GRPO keeps online sampling and the clipped surrogate but removes the critic: for each prompt it draws a group of completions, uses the group’s mean reward as the baseline, and z-scores the rewards to get one scalar advantage that is broadcast to every token of its completion. So the axis is not “better versus worse” but which piece of machinery you are willing to pay for: PPO buys fine-grained credit assignment with a learned value function, DPO buys simplicity by giving up exploration, and GRPO buys on-policy learning with a Monte Carlo baseline.

(1) Networks Resident: PPO needs policy, reference, reward model, and critic; GRPO drops the critic; DPO drops both the critic and the reward model.
(2) Advantage Estimation: PPO uses GAE over a learned , GRPO uses a group-relative z-score of sequence rewards, and DPO never forms an advantage at all, only a reward margin between two responses.
(3) On-Policy Versus Offline: PPO and GRPO resample from the current policy every iteration, so the clip and importance ratio are meaningful; DPO trains on a static dataset and is therefore exposed to distribution shift.
(4) KL Control: PPO and GRPO add an explicit KL penalty (GRPO commonly uses the low-variance k3 estimator), while DPO’s KL constraint is baked into the log-ratio parameterization and controlled solely by .
(5) Credit Granularity: only PPO assigns different advantages to different tokens; GRPO gives every token in a completion the same scalar, and DPO gives a whole-sequence gradient.
(6) Where Each Fits: verifiable rewards from a checker or unit test favor GRPO, a cheap single-pass alignment on collected preferences favors DPO, and a nuanced learned reward model with long generations favors PPO.

Figure 1: The three objectives differ mainly in what sits between the policy and the loss: PPO inserts a reward model plus a trained critic, GRPO replaces the critic with a group of sampled rollouts, and DPO removes the sampling stage so the frozen reference is the only extra network.

DPO’s derivation is what makes the contrast precise. The KL-constrained bandit objective has the closed-form optimum ; solving for the reward gives , and because the Bradley-Terry likelihood depends only on reward differences for the same prompt, the intractable partition function cancels. The reward model therefore never has to be materialized, since its optimal policy is the object you were going to train anyway. The price is that this equivalence is exact only when the preference pairs come from ; on off-policy pairs the loss can raise the margin while pushing down the probability of the chosen response as well, the failure mode usually called likelihood displacement. GRPO takes the opposite trade, keeping the on-policy ratio and clip but swapping the critic’s learned variance reduction for a Monte Carlo baseline over samples of the same prompt, which is cheap and well-behaved when the reward is a verifiable 0/1 signal from a math checker or unit test. That is exactly the regime DeepSeek used when introducing GRPO for DeepSeekMath and then scaling it for R1.

Mathematical Formulation:

Where:

• is the trained policy, the frozen reference (normally the SFT checkpoint), and the policy that generated the current batch of rollouts.
• is the prompt-plus-prefix state and the token emitted at position ; is the per-step reward, which in RLHF is usually nonzero only at the final token.
• is the learned critic, the TD residual, and the GAE advantage with discount and trace decay .
• is the clip half-width (typically 0.1 to 0.2) and is the KL coefficient in PPO and GRPO, or the implicit-reward temperature in DPO (typically 0.01 to 0.5).
• and are the preferred and rejected responses for prompt , is the logistic function, and is the reward implied by the policy itself.
• indexes the group of completions sampled per prompt (commonly to ), is its sequence-level reward, and is shared by every token of completion .
• Required initial condition in all three: is initialized from , otherwise the KL term and the log-ratio reward have no meaningful anchor.

Figure 2: Counting only weights at 7B scale in bf16, PPO holds four copies and trains two of them, so its Adam state is roughly double GRPO’s; a rule-based verifier removes the reward model entirely, making GRPO as light as DPO in weight memory while still sampling online.