ENID
Lensa ML

RLHF

Step 1 of 6: The Alignment Problem

Why raw language models need alignment

Temperature1
Token probability distributionThe35%cat19%sat13%on12%the8%mat5%is4%a4%Generated: the on the cat the The
Temperature
1.00
Top prob
34.6%
Entropy
2.63 bits

Moderate temperature: the samples read fluently, but nothing in sampling makes them helpful, honest, or harmless.

Drag the temperature from 0.3 to 2.0 and watch the entropy readout rise. Temperature divides the logits before softmax, p_i = exp(z_i / T) / Σ exp(z_j / T), so it only reshapes a distribution the model has already learned. Pretraining teaches a model to predict the next token of web text, which means it imitates what usually comes next rather than what a user asked for. InstructGPT showed how large that gap is: labelers preferred a 1.3B aligned model over the 175B GPT-3 it started from. The usual first fix is supervised fine-tuning (SFT) on written demonstrations.

Step 2 of 6: Collecting Human Preferences

Building a dataset of ranked outputs

Comparisons4
Human preference pairsAConcise and accurate answer✓BRambling off-topic answerAVerbose but correct replyBClear helpful explanation✓AShort but misleading textBCorrect but rude response✓AHelpful step-by-step guide✓BIncomplete vague response
Pairs
4
Dataset size
8 responses
Human hours
2.0 est.

Moderate dataset: enough pairs to capture basic preferences. Annotator quality and diversity matter as much as count.

Drag Comparisons from 1 to 10 and watch the dataset size and human-hours estimate grow. Labelers compare answers instead of scoring them because picking the better of two is faster and more consistent across people than assigning a grade. InstructGPT had labelers rank 4 to 9 responses per prompt. A ranking of K responses gives K(K − 1)/2 pairs, so one ranking of 9 yields 36 comparisons. Its reward model was trained on about 33,000 prompts. Labelers in that study agreed with each other about 73% of the time, which caps how accurate any reward model can be.

Step 3 of 6: Training a Reward Model

Learning to score outputs like humans

Train Step0
Reward scoresHelpful, accurate answer0.07Polite but wrong reply0.32Rude but correct text0.20Harmful misleading output0.12Clear step-by-step guide0.28Accuracy (illustrative)1.00.5
Step
0
Accuracy
50.0%
Best score
0.32

Early training: all five scores sit near the same value plus noise. The model cannot yet tell good outputs from bad.

Drag Train Step from 0 to 20 and watch the scores spread apart while the accuracy curve climbs. The reward model is usually a copy of the language model with its output layer swapped for one number. Each pair is trained with the Bradley-Terry loss, L = −log σ(r(x, y_w) − r(x, y_l)), where y_w is the chosen answer and y_l the rejected one. The loss only sees the gap, so rewards have no fixed zero. The curve levels off near 72%, about where real reward models land on held-out human comparisons and close to the labelers' own agreement.

Step 4 of 6: PPO Optimization

Updating the policy with reward signal

KL weight β0.50
Base (blue) vs policy (purple) distributionT0T1T2T3T4T5T6T7BasePolicy (PPO)
KL weight
0.50
KL div
0.498
Reward
0.818
Objective
0.569

Moderate β: reward improves while the policy stays near the base model, which avoids degenerate text. The purple bars are the exact best policy for this β, π ∝ π_base · exp(r/β), so no other distribution gets a higher objective. PPO works toward it by sampling responses and taking clipped gradient steps.

Drag the KL weight β from 0.05 to 2 and compare the purple policy bars with the blue base bars. PPO maximizes r(x, y) − β · KL(π || π_ref): the reward model's score minus a penalty for moving away from the frozen reference model. PPO (Schulman et al., 2017) is a policy-gradient method that clips each update so the policy cannot change too much from one batch of samples. Each full response counts as one episode, scored once at the end. Training keeps four models in memory: the policy, the reference, the reward model, and a value model.

Step 5 of 6: KL Divergence Constraint

Keeping the model close to its base

KL Budget1
Base model vs aligned modelABCDEFBaseAligned
KL budget
1.0
KL divergence
1.000
Constraint
Binding

Moderate KL budget: enough room to change behavior while the model keeps its core capabilities. The teal bars are the best policy within the budget: π ∝ π_base · exp(r/β), with β set so the KL equals the budget.

Drag the KL budget from 0.1 to 5 and watch the teal aligned bars pull away from the blue base bars. KL(P || Q) = Σ P(x) log(P(x) / Q(x)) is zero when the two distributions match and grows as P puts mass where Q puts little. Without this limit the policy finds text that the reward model scores highly but people dislike, a form of Goodhart's law. Some implementations set a target KL and tune β to hit it, close to this budget view. InstructGPT also mixed in pretraining gradients (PPO-ptx) to protect benchmark scores.

Step 6 of 6: DPO Alternative

Direct preference optimization without a reward model

DPO β0.10
PPO vs DPO pipelinePPO (4 models in memory)SFT ModelReward ModelPPO PolicyDPO (2 models)Direct OptimizationPreference margins (chosen − rejected)Helpful answervs Harmful output0.602.10Clear explanationvs Confusing reply1.202.20Honest responsevs Deceptive text0.102.10ReferenceDPO Policy
DPO β
0.10
Avg margin
2.13
Shift vs ref
1.50

Low β: DPO follows the preferences hard and drifts far from the reference model. Simpler than PPO, with the same reward hacking risk. PPO keeps four models in memory: the policy, a frozen reference copy of the SFT model, the reward model, and a value model. DPO needs two: the policy and the frozen reference.

Drag DPO β from 0.05 to 0.5 and compare the reference margins with the policy margins. DPO (Rafailov et al., 2023) shows that the KL-constrained objective has a closed-form best policy, so the reward can be written through the policy itself: r(x, y) = β log(π(y|x) / π_ref(y|x)), plus a term that cancels when two answers are compared. The Bradley-Terry loss then becomes a plain classification loss on preference pairs, with no sampling and no reward model. A reward gap is β times the policy's shift from the reference margin, so a smaller β needs a larger shift: 1.5 on average at the default of 0.1, and 0.3 at 0.5. Zephyr and Llama 3 were tuned with DPO.

What is RLHF (reinforcement learning from human feedback)?

Reinforcement learning from human feedback (RLHF) is a way to fine-tune a model using human judgments of its outputs instead of hand-written correct answers. People compare pairs of responses, a reward model learns to predict which one they prefer, and the language model is then trained with reinforcement learning to produce responses that the reward model scores highly.

RLHF is the step that turned large pretrained language models into usable assistants. OpenAI's InstructGPT (2022) used it to make GPT-3 follow instructions, and the same recipe was used for ChatGPT. It is applied wherever the goal is easy to judge but hard to write down as a loss: helpfulness, tone, refusing harmful requests, and summary quality. Direct preference optimization (DPO) is a simpler variant that many open models now use.

Key ideas

  • RLHF fine-tunes a model toward outputs people prefer, using comparisons instead of written answers.
  • The standard pipeline is supervised fine-tuning, then a reward model, then reinforcement learning with PPO.
  • A reward model trained on pairs learns only reward differences, through the loss −log σ(r_chosen − r_rejected).
  • A KL penalty against the reference model keeps the policy from drifting into text that games the reward model.
  • Reward hacking is when the learned reward goes up but human judgments of the output do not.
  • DPO trains directly on preference pairs and needs no separate reward model or RL loop.

Common questions

What is the difference between RLHF and supervised fine-tuning?

Supervised fine-tuning trains on demonstrations: the model copies a written answer token by token. RLHF trains on judgments of the model's own outputs, so it can improve behavior that people can recognize but find hard to write out. Most pipelines run SFT first and RLHF after it.

Why is it called reinforcement learning?

The model is treated as a policy. It generates a full response, receives a single scalar reward, and is updated to make high-reward responses more likely. There is no correct target token at each position, only a score for the whole answer, which is the setting reinforcement learning is built for.

Where did RLHF come from?

Christiano et al. (2017) trained agents in Atari games and simulated robotics from human comparisons of short video clips. With about 900 comparisons, a simulated robot learned to do a backflip. The method was later applied to text summarization and then to instruction following in InstructGPT.

Is RLHF still used, or has DPO replaced it?

Both are used. DPO is cheaper and simpler to run, so many open-weight models use it or one of its variants. PPO-based RLHF is still used when a separate reward model is wanted, for example to score fresh samples during training. Many pipelines also replace some human labels with AI-generated feedback (RLAIF).

Related explainers

References

  1. Ouyang et al. (2022), "Training language models to follow instructions with human feedback" (InstructGPT)
  2. Christiano et al. (2017), "Deep reinforcement learning from human preferences"
  3. Rafailov et al. (2023), "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"
  4. Schulman et al. (2017), "Proximal Policy Optimization Algorithms"

Last updated