Design an RLHF alignment pipeline for aligning a large language model with human preferences. A foundation model team must align a model so that it follows instructions, refuses harmful requests, and produces helpful, harmless, and honest responses.
The alignment must use reinforcement learning from human feedback, but the team must choose between PPO (Proximal Policy Optimization), GRPO (Group Relative Policy Optimization), and GSPO (Group Sequence Policy Optimization) as the alignment algorithm. Each has different sample efficiency, training stability, and compute requirements. The system must train a reward model from human preference data, run the alignment loop at scale on a distributed cluster, and produce a model that passes safety evaluations while keeping its capability.
How would you design this system? Cover the reward modeling architecture, the candidate RL algorithms (PPO vs GRPO vs GSPO) and their trade-offs, how you handle reward hacking and mode collapse, how you distribute the alignment training across GPUs, and how you evaluate alignment quality.
The Problem: humans can only tell you which of two whole responses they prefer, but gradient descent needs a scalar objective and a stable update signal at every token of every rollout.
Answer
The stack is the standard three stages: an instruction-tuned SFT checkpoint, a Bradley-Terry reward model trained on human preference pairs, and an on-policy RL loop that maximizes reward minus a KL penalty against the frozen SFT reference. Everything above the update rule is shared, so the real design question is the RL objective. PPO learns a separate value network to supply per-token baselines, while GRPO and GSPO delete the critic and use the mean reward of a group of sampled responses instead, removing one trainable model copy from memory. GSPO goes further and computes the importance ratio at the sequence level, matching the granularity of the reward itself, which is what keeps training stable on MoE policies and long rollouts. My default is group-relative advantages with a sequence-level ratio, a KL anchor plus reward-model ensembling to hold back reward hacking, and PPO kept in reserve only where a dense token-level value estimate genuinely earns its extra trainable model.
(1) Shared Substrate: SFT policy, preference-trained reward model, and a frozen reference for the KL anchor; all three candidates optimize the same penalized reward.
(2) PPO: actor-critic; a learned value network produces per-token advantages through GAE, and the importance ratio is clipped once per token.
(3) GRPO: no critic; sample 
responses per prompt and standardize their rewards into one advantage per response, still clipped per token.
(4) GSPO: same group baseline, but the importance ratio and the clip act on the whole length-normalized sequence, matching the sequence-level reward.
Figure 1: One alignment loop: preferences train the reward model, reward minus a KL anchor scores group rollouts, and only the update box changes between PPO, GRPO, and GSPO.
Clarify Before Designing:
(1) Policy Architecture: dense or sparse MoE, and at what size? Routing volatility in MoE is the single strongest argument for a sequence-level ratio.
(2) Reward Source: learned reward model only, or are parts of the traffic verifiable (unit tests, math checkers, format rules) so a rule-based reward can replace the model?
(3) Sequence Length: 1k-token chat replies or 32k-token reasoning traces? Long rollouts multiply token-level ratio variance and dominate step time.
(4) Compute Budget: how many GPUs, and does a second trainable critic-sized model fit next to the policy after sharding?
(5) Preference Data: how many pairs exist, what is the annotator agreement rate, and is there a recurring human evaluation loop or only an LLM judge?
(6) Release Bar: what refusal rate on red-team prompts and what maximum capability regression (the alignment tax) gate the release?
Leave a Reply