DL0175 RLAIF and Constitutional AI

RLAIF keeps every moving part of RLHF and swaps only the label source. Instead of paying annotators to rank two candidate responses, a frozen critic (feedback) model is shown the prompt, both candidates, and a short natural-language principle, and it answers a multiple-choice question whose option log-probabilities become the preference label. Constitutional AI is the best-known instantiation and runs in two stages: a supervised stage where the model repeatedly critiques and revises its own harmful answers against principles sampled from a written constitution, producing the SL-CAI checkpoint, and a reinforcement stage where AI-generated harmlessness comparisons train a preference model that then drives PPO, producing RL-CAI. The scaling change is blunt: a comparison that costs roughly a dollar and minutes of human attention becomes two forward passes, so label volume becomes a compute decision rather than a hiring decision, and labels arrive as calibrated soft probabilities rather than single bits. The alignment change is subtler and more interesting, because the optimization target moves from an implicit, unauditable aggregate of annotator taste into a short document you can read, diff, and version, and because self-critique lets a model be harmless without collapsing into evasive refusals.

(1) Only The Labeler Changes: the preference-model loss, the KL-regularized PPO objective, and the reward-hacking dynamics are identical to RLHF, so RLAIF is not a new algorithm but a new data supply.
(2) Two Stages In Constitutional AI: critique → revision → SL-CAI supervised finetuning, then AI-labeled comparisons → preference model → RL-CAI, with SL-CAI serving as both policy initialization and KL reference.
(3) One Principle Per Judgment: a single principle is sampled from the constitution for each critique and each comparison, which keeps prompts short and spreads coverage across the document instead of demanding the model weigh 16 rules at once.
(4) Soft Labels, Not Bits: the critic emits from option token logits, which carries more information per comparison than a human click and lets you distil graded preference strength.
(5) Auditable Objective: a behavior change becomes a text edit to the constitution plus a rerun, rather than a new annotation guideline, a new labeler cohort, and a multi-week collection cycle.
(6) Bounded By The Critic: AI feedback inherits the critic’s position bias, self-preference, and capability ceiling, so it is strongest on axes where judging is easier than generating and weakest on novel or expert domains.

Figure 1: The constitution is the only human-written artifact in the loop, and it touches the pipeline twice: once to steer self-critique and revision in the supervised stage, once to condition the critic’s pairwise judgments in the RL stage. Everything downstream of the label, the preference model and the KL-penalized policy update, is unchanged RLHF.

Three implementation details do most of the real work. First, order debiasing: language models judging “(A) or (B)” have a strong position preference, so each pair is scored twice with the options swapped and the two probabilities are averaged, which is cheap because compute is the only cost. Second, chain-of-thought judging with clamped probabilities: letting the critic reason before answering improves agreement with humans but produces near-deterministic 0.0 or 1.0 preferences, so the CAI work clamps the resulting probabilities into a narrow band around 50 percent to keep the preference model from training on overconfident targets. Third, a deliberately mixed label supply: RL-CAI does not eliminate humans, it reassigns them, training the preference model on human helpfulness comparisons alongside AI harmlessness comparisons, because harmlessness is the axis where labeling is unpleasant, high-variance, and easy to specify in writing. Empirically, the reported effect is not merely cheaper labels but a better trade-off, with RL-CAI models both more harmless and less evasive than an RLHF baseline, while the RLAIF study on summarization found AI-labeled and human-labeled policies statistically indistinguishable to human raters despite the critic agreeing with humans only about 78 percent of the time.

Mathematical Formulation:

Where:

• is the response after revisions of the original answer to red-team prompt , and is the helpful-only model doing both the critique and the rewrite.
• is one principle sampled uniformly from the constitution , typically around 16 short written rules.
• and are the critic’s logits for the option tokens of the multiple-choice question, and is the logistic function, so the label is a probability rather than a bit.
• and are the probabilities assigned to the same candidate under the two presentation orders, and is the order-debiased label actually stored.
• is the preference model, and the winning and losing responses under , and the standard Bradley-Terry loss, unchanged from RLHF.
• is the policy being optimized, is the frozen SL-CAI checkpoint, and sets how far the policy may drift before the KL penalty dominates the reward.

Label Volume At A Fixed 100k USD Budget:

The numbers assume a fully loaded cost near 0.75 USD per human comparison against roughly 0.002 USD for two short critic forward passes, which is why the same budget buys about 375 times more comparisons. Exact rates move with model size, context length, and vendor pricing, but the two orders of magnitude are robust, and they change what is even worth attempting: per-principle coverage sweeps, full relabeling after a constitution edit, and comparison sets larger than any human dataset ever collected. The binding constraint stops being annotation throughput and becomes critic quality plus reward-model overoptimization.

Figure 2: Two different claims. On the left, the cost curve is the scaling argument, and at a fixed budget it is worth about 375 times more comparisons. On the right, the frontier shift is the alignment argument: human feedback tends to trade helpfulness against harmlessness, while self-critique plus AI harmlessness labels moves the operating point off that line instead of sliding along it. Right-panel positions are schematic illustrations of the reported qualitative result, not measured Elo scores.