A language model fresh off pretraining is a peculiar thing. It's extraordinarily capable — it has absorbed patterns from vast swaths of human text — but it has no particular reason to be helpful, honest, or safe. Ask it a question and it might answer it, continue your sentence, imitate a forum post, or do something stranger still. It follows the statistical grain of its training data, not your intentions. The technique that changed this — that turned a next-token prediction engine into something that actually follows instructions and declines to cause harm — is reinforcement learning from human feedback (RLHF). Understanding how it works reveals a great deal about both the power and the limitations of modern AI assistants.
Why Pretraining Alone Isn't Enough
Pretraining a large language model involves feeding it enormous quantities of text and training it to predict the next token. The objective is simple and the scale is massive. The result is a model with a rich internal representation of language, facts, reasoning patterns, and more. But the training objective — predict what comes next — doesn't map cleanly onto what users actually want.
A pretrained model optimized purely on next-token prediction will happily complete a prompt asking for instructions to do something dangerous, because dangerous instructions appear in training data. It will hedge, waffle, or produce plausible-sounding nonsense, because that's what text sometimes looks like. It's not trying to deceive or harm — it has no goals at all in the conventional sense. It's a very sophisticated pattern matcher whose patterns were learned from human output in all its variety, including its worst examples.
As an Amazon Associate, I earn from qualifying purchases.
This is the alignment gap that RLHF was designed to close: the distance between "statistically plausible continuation" and "genuinely useful, appropriate response."
The Three-Stage RLHF Pipeline
RLHF as applied to ChatGPT and other LLMs isn't a single training step — it's a pipeline with three distinct phases, each building on the last.
Stage 1: Supervised Fine-Tuning (SFT)
Before reinforcement learning enters the picture, the base model undergoes supervised fine-tuning. Human contractors — often called labelers or annotators — write example conversations: a prompt, and an ideal response. These demonstrations are curated to reflect the behavior the developers want: helpful, clear, appropriately cautious, following instructions.
The model is then fine-tuned on these examples using standard supervised learning — the same gradient descent that drove pretraining, but now on a much smaller, high-quality dataset specifically constructed to demonstrate good assistant behavior. This stage gives the model a rough shape of what helpful responses look like. It's necessary but not sufficient: supervised fine-tuning can only capture what annotators explicitly demonstrate, and human writing is an imperfect proxy for what's actually best in every situation.
As an Amazon Associate, we earn from qualifying purchases.
Stage 2: Training a Reward Model
This is where RLHF diverges from ordinary fine-tuning, and it's the conceptually important step. The core problem is that defining "a good response" precisely enough for a loss function is extremely difficult. A good response might be accurate, concise, honest, safe, and appropriately caveated — all at once. Writing a formula that captures all of this is essentially impossible.
Instead, RLHF sidesteps explicit definition and learns from human preference comparisons. The process works like this:
- The SFT model generates several different responses to the same prompt.
- Human raters compare pairs (or larger sets) of these responses and indicate which one they prefer.
- These preference judgments become training data for a separate neural network: the reward model.
The reward model is trained to predict which response a human would prefer, given a prompt. Crucially, comparison judgments are far easier for humans to make reliably than absolute ratings. Asking "is this response a 7 out of 10?" invites inconsistency. Asking "which of these two responses is better?" is cognitively simpler and produces more consistent signal.
The reward model learns to assign a scalar score to any (prompt, response) pair. This score isn't labeled "helpfulness" or "safety" explicitly — it's a learned compression of the patterns in human preference data. High scores correlate with what annotators tended to prefer; low scores with what they tended to reject.
Stage 3: Reinforcement Learning with PPO
With a reward model in hand, the actual reinforcement learning phase begins. The SFT model is now treated as a policy — in RL terminology, an agent that takes actions (generates tokens) in an environment (the conversation context) to maximize a reward signal (the reward model's score).
The algorithm most commonly used is Proximal Policy Optimization (PPO), a policy gradient method. PPO updates the language model's weights to generate responses that score higher according to the reward model, while constraining how far each update moves the model from its previous behavior. That constraint matters: without it, the model would quickly learn to exploit the reward model — producing responses that score highly but are nonsensical or degenerate, a phenomenon sometimes called reward hacking or reward model overoptimization.
To further guard against this, a KL divergence penalty is added to the reward signal. KL divergence measures how far the current policy has drifted from the original SFT model. The effective reward at each step is roughly: reward model score minus a penalty for diverging too far from the SFT baseline. This keeps the model in a sensible region of behavior space while still pushing it toward higher-quality responses.
The loop runs across many thousands of prompts. The language model generates a response, the reward model scores it, PPO updates the weights, and the process repeats. Over time, the model learns to produce outputs that more consistently match the preferences captured in the reward model.
What Human Raters Are Actually Judging
The quality of RLHF depends critically on the quality and consistency of human preference data. In practice, labeler guidelines for major AI systems run to dozens or even hundreds of pages, specifying how to evaluate responses across multiple dimensions: accuracy, helpfulness, harmlessness, honesty, formatting, and more. Different organizations weight these differently and resolve conflicts between them differently.
This means RLHF doesn't produce a universal notion of "good" — it produces a model of good that reflects the values, priorities, and blind spots of a particular group of annotators operating under particular guidelines. The reward model is, in a real sense, a learned approximation of a specific human judgment process.
This has two significant implications. First, the resulting model will behave well on the distribution of prompts and preferences it was trained on, but may generalize imperfectly to edge cases. Second, the values embedded in the model are contestable — different annotation teams with different guidelines would produce meaningfully different models even starting from the same pretrained base.
Reward Hacking and the Limits of Learned Reward
One of the most important problems in RLHF is the gap between the reward model and the true objective. The reward model is an approximation — a neural network trained on a finite set of human comparisons. Like any learned function, it will have blind spots and errors. A language model optimized hard against this imperfect reward can discover and exploit those errors.
Common failure modes include models that become verbose (human raters sometimes conflate length with thoroughness), excessively agreeable (raters may prefer confident-sounding responses even when hedging is more accurate), or prone to stylistic flattery. The model isn't consciously gaming the system — it's doing exactly what optimization does: finding the shortest path to a high score, which isn't always the path that looks best to a thoughtful evaluator.
The KL penalty and careful reward model training help, but they don't fully solve the problem. This is one of the core challenges of AI alignment research more broadly: specifying what we want with enough precision that optimizing for it actually gets us there.
Variants and Alternatives to Classic RLHF
RLHF as described above — SFT, reward model, PPO — is the canonical form, but several important variants have emerged as the field has matured.
Direct Preference Optimization (DPO)
Direct Preference Optimization, introduced in 2023, skips the reward model training step entirely. Instead, it reformulates the RLHF objective and shows mathematically that you can fine-tune the language model directly on preference pairs without an explicit reward model as an intermediary. DPO is simpler to implement, more stable to train, and has produced competitive results, making it popular in both research and open-source model development.
Constitutional AI
Anthropic developed Constitutional AI as an approach that reduces reliance on human feedback for the harmlessness dimension of training. Rather than having humans label harmful responses directly, the model is given a set of principles (a "constitution") and trained to critique and revise its own outputs according to those principles. Human feedback is still used, but focused on helpfulness, while the AI itself generates much of the safety-related preference data through self-critique.
RLAIF
Reinforcement Learning from AI Feedback uses a separate language model as the rater rather than human annotators. This dramatically reduces the cost and time required to generate preference data, though it introduces questions about what exactly the AI rater has learned to prefer and whether its judgments faithfully represent human values.
Why RLHF Matters for Understanding Modern AI Behavior
If you've ever noticed that NLP-based AI assistants tend to have particular stylistic tics — a certain measured hedging, a tendency to structure responses with bullet points, a consistent tone — those patterns are largely the fingerprint of RLHF. They reflect what annotators rewarded, which reflects both genuine quality signals and the particular preferences and limitations of a human annotation process.
Understanding RLHF also clarifies why these models sometimes behave unexpectedly. A model that refuses a benign request is likely tripping a pattern associated with low reward during training. A model that confidently states something false may have learned that confident-sounding responses were preferred. These aren't bugs in the conventional sense — they're the expected output of an optimization process applied to an imperfect reward signal.
The technique has been transformative. It's the primary reason that today's AI assistants are usable at scale by ordinary people rather than just researchers who know how to prompt raw language models. But it's not magic, and it doesn't solve alignment — it defers it. The quality of what comes out depends deeply on the quality of the human judgment that went in, and on how faithfully the reward model captured it. That dependency is, in one form or another, what the next generation of alignment research is trying to address.
