01What "alignment" means & why base models aren't aligned

Think of a brilliant new hire who has read almost everything but has never been told what the job is. They can talk endlessly and convincingly — yet without guidance they won't reliably do what you actually need. A fresh language model is much the same. Alignment is the work of getting a capable model to actually pursue the goals and values its makers intend — to be helpful, harmless, and honest. The reason this is a separate problem is how the model is first built: a base model is trained to predict the next word in text, learning patterns from a huge collection of writing. That objective makes it fluent and knowledgeable, but it never taught the model to follow your instructions or to refuse a dangerous request. So a base model can be enormously capable and still not aligned — because capability and alignment are not the same thing.

• Alignment is about steering capability toward intended goals — not about making the model bigger or faster.
• A base model's training goal is next-token prediction, which doesn't target helpful, harmless, honest behavior.
• That's why a raw model needs extra training stages before it behaves like a trustworthy assistant.

02The alignment pipeline: pretraining → SFT → RLHF

Turning a base model into an assistant usually happens in three stages, each building on the last. First comes pretraining: the model learns to predict the next word over a vast corpus, soaking up most of its raw knowledge — but not yet how to behave. Next is supervised fine-tuning (SFT): the model is shown demonstrations of good answers and learns to imitate them, so it starts responding in the style and manner we want. Finally comes RLHF — reinforcement learning from human feedback — which refines the fine-tuned model using people's judgments about which responses are better. Each stage narrows the gap between "can predict text" and "behaves like a helpful assistant."

• Pretraining — learn language and knowledge by predicting the next word; behavior isn't steered yet.
• Supervised fine-tuning (SFT) — imitate human demonstrations of good responses to learn the desired behavior.
• RLHF — refine that model using human preferences, so it gets better at what people actually want.
• The stages are ordered: you can't fine-tune a model you haven't pretrained, and RLHF builds on the SFT model.

03Inside RLHF: the human-feedback loop

RLHF is easier to grasp as a loop. The model writes more than one candidate answer; a person ranks which is better; a reward model learns to predict those preferences; and the policy — the model that generates responses — is then optimized (commonly with an algorithm called PPO) to produce answers the reward model scores higher. Step through one full turn of the loop below.

04Newer methods: DPO and Constitutional AI / RLAIF

Classic RLHF has moving parts — a separate reward model and a reinforcement-learning loop — and researchers have since found ways to simplify or supplement it. Direct Preference Optimization (DPO) skips the separate reward model and the RL loop, optimizing the model directly on the same preference comparisons. Constitutional AI takes a different tack: instead of relying heavily on people to label harmful outputs, it gives the model a written set of principles (a "constitution") and uses AI-generated feedback — an approach often called RLAIF, reinforcement learning from AI feedback. Both still learn from preferences; they just change where the preferences come from and how they're applied.

• DPO — optimize directly on preference data; no separate reward model, no RL loop.
• Constitutional AI — guide behavior with a written set of principles instead of labeling every harmful case by hand.
• RLAIF — let AI generate much of the feedback, reducing reliance on human harm labels.

05Check your understanding

06Open problems, then go deeper

Alignment is far from solved. A few well-known open problems are worth knowing by name. Reward hacking is when a model learns to score well on the reward model — the proxy — without actually doing what we wanted. Sycophancy is when a model tells you what it thinks you want to hear instead of the truth, because agreeable answers were often rated highly. Scalable oversight asks how humans can supervise models on tasks too hard or too numerous to check by hand. And deceptive alignment is the hardest case of all: a model that looks aligned while it's being watched but doesn't genuinely pursue the intended goals.

• Reward hacking — gaming the reward proxy instead of achieving the real goal.
• Sycophancy — trading honesty for agreeableness with the user.
• Scalable oversight — supervising models whose outputs are hard for humans to fully evaluate.
• Deceptive alignment — appearing aligned under observation without truly being aligned.