01Why next-word prediction isn't enough

A base language model is trained on one job: predict the next token across a huge pile of internet text. That makes it fluent, but fluent isn't the same as useful. Ask a base model a question and it may continue with more questions, drift off-topic, or produce confident nonsense — because imitating text is not the same as following your intent. RLHF is the technique that re-aims the model from "what text usually comes next" toward "what a person would actually prefer as a response." It does this by collecting human judgments about which outputs are better, turning those judgments into a numeric reward, and then optimizing the model to earn more of that reward.

• The goal is alignment with human preferences — making outputs more helpful, honest, and harmless, and better at following the user's intent than pretraining alone (Ouyang et al., InstructGPT).
• Strikingly, InstructGPT showed a 1.3B-parameter RLHF-tuned model could be preferred by labelers over the 175B-parameter GPT-3 on their prompt distribution — alignment, not just size, drives perceived quality.
• Those preference results are measured on specific prompts and labeler pools; treat them as evidence of better alignment, not a universal accuracy guarantee.

02The three-stage pipeline, step by step

The canonical RLHF recipe has three stages: (1) supervised fine-tuning on example demonstrations, (2) train a reward model from human comparisons between candidate responses, and (3) policy optimization — usually with PPO — that nudges the model toward higher-reward outputs while a KL penalty keeps it from drifting too far. Step through each stage. In stage 2 you'll rank responses yourself to teach the reward model; in stage 3 you'll run optimization steps and watch the model's output distribution shift toward what you preferred.

03The reward model: turning preferences into a number

Reinforcement learning needs a reward — a single number that says how good an action was. But "how good is this paragraph?" has no obvious score. The trick is to not ask for a score at all. Instead, humans are shown two (or more) candidate responses to the same prompt and simply pick which one they prefer. A reward model is then trained on thousands of these comparisons to predict which response a human would choose — and in doing so it learns to output a scalar reward for any response. The idea predates LLMs: Christiano et al. (2017) learned reward models from human preferences over pairs of trajectory segments for control tasks, asking for feedback on well under 1% of the agent's interactions.

• Humans give comparisons, not scores — "A is better than B" is far more reliable and consistent than asking people to rate things 1–10.
• The reward model generalizes the human's taste: once trained, it can score brand-new responses the labelers never saw.
• The reward model is imperfect. It can be gamed (reward hacking) and it inherits the values and biases of the specific labeler pool — its preferences are relative to those raters, not an objective standard.

04Optimization: chase reward, but stay on a leash

With a reward model in hand, the final stage uses reinforcement learning to adjust the model's weights so its outputs earn higher reward. The most common algorithm is Proximal Policy Optimization (PPO) (Schulman et al., 2017), a clipped policy-gradient method that makes cautious, bounded updates instead of large risky ones. There's a catch: if you optimize for the reward model alone, the policy can drift into weird, degenerate text that scores high on the imperfect reward but reads badly — classic reward over-optimization. The fix is a KL-divergence penalty against a frozen reference model (usually the SFT model), introduced for language-model fine-tuning by Ziegler et al. (2019). It acts as a leash: chase reward, but don't wander too far from sensible language.

• PPO updates are clipped / bounded — small, careful steps rather than big jumps that could destabilize the model.
• The KL penalty trades reward against staying close to the reference model; tuning that balance is central to stable RLHF training.
• RLHF is known to be complex and sometimes unstable, and an "alignment tax" can regress performance on some tasks — present it as a powerful but delicate technique, not a free win.

05Beyond the textbook recipe

The SFT → reward model → PPO pipeline is the historical reference, but production systems mix and match. DPO drops the reward model and RL loop, optimizing preferences directly. RLAIF / Constitutional AI (Bai et al., 2022, Anthropic) substitutes AI-generated feedback for some human harm labels: a model critiques and revises its own responses against a written set of principles (a "constitution"). Lighter-weight options like reward-ranked / best-of-n fine-tuning (RAFT) and ranking-loss methods (RRHF) avoid a full online RL loop while still using preference signals. The constant across all of them is the core idea: human (or human-derived) preferences shape the model's behavior.

• DPO — preference pairs, no reward model, no RL loop; a simpler, increasingly common path.
• RLAIF / Constitutional AI — AI feedback against written principles replaces some human labels, scaling the feedback step.
• RAFT / RRHF — reward-ranked or ranking-loss alignment that sidesteps full online RL; treat all of these as active alternatives, not a single settled "best" method.

06Check your understanding

07Take it with you & go deeper