RLHF
The Alignment Problem
Why raw language models need alignment
Moderate temperature: the model samples fairly well, but there's no guarantee the output is helpful, honest, or harmless. We need alignment.
Step 1 of 6: The Alignment Problem
Why raw language models need alignment
Moderate temperature: the model samples fairly well, but there's no guarantee the output is helpful, honest, or harmless. We need alignment.
Step 2 of 6: Collecting Human Preferences
Building a dataset of ranked outputs
Moderate dataset: enough pairs to capture basic human preferences. Quality and diversity of annotators matter as much as quantity.
Step 3 of 6: Training a Reward Model
Learning to score outputs like humans
Early training: the reward model assigns near-random scores. It hasn't learned to distinguish good outputs from bad ones yet.
Step 4 of 6: PPO Optimization
Updating the policy with reward signal
Moderate β: a good balance. The policy improves on reward while staying close to the base model, avoiding degenerate behavior.
Step 5 of 6: KL Divergence Constraint
Keeping the model close to its base
Moderate KL budget: enough room for meaningful alignment while keeping the model's core capabilities intact. This is the typical operating range.
Step 6 of 6: DPO Alternative
Direct preference optimization without a reward model
Low β: DPO aggressively follows preferences, deviating far from the reference model. Simpler than PPO but same reward hacking risks.