What is KL divergence?
KL divergence (Kullback-Leibler divergence) measures how much a probability distribution Q differs from a reference distribution P. It is defined as KL(P‖Q) = Σ P(x) log(P(x)/Q(x)), the expected extra information needed to encode samples from P with a code built for Q. It is zero exactly when P and Q are identical and positive otherwise, and it is not symmetric.
KL divergence appears all over machine learning. Training a classifier with cross-entropy loss minimizes the KL divergence from the label distribution to the model's predictions. VAEs add a KL term that keeps the encoder's latent distribution close to a prior, knowledge distillation trains a small model to match a large model's output distribution, and RLHF penalizes a fine-tuned language model for drifting too far, in KL, from its reference model.
Key ideas
- KL(P‖Q) = Σ P(x) log(P(x)/Q(x)) is the expected log ratio of P to Q under P.
- KL divergence is never negative, and it is zero only when P = Q.
- KL(P‖Q) and KL(Q‖P) usually differ, so KL is not a distance metric.
- Cross-entropy equals entropy plus KL divergence: H(P, Q) = H(P) + KL(P‖Q).
- Forward KL(P‖Q) is mode-covering, and reverse KL(Q‖P) is mode-seeking.
- With one-hot labels, cross-entropy loss and KL divergence are the same number.
Common questions
Who introduced KL divergence?
Solomon Kullback and Richard Leibler defined it in their 1951 paper "On Information and Sufficiency", building on Shannon's information theory. They called it the mean information for discrimination between two hypotheses. In statistics it is also known as relative entropy.
What is the KL divergence between two Gaussians?
For one-dimensional normals it has a closed form: KL(N(μ₁, σ₁²) ‖ N(μ₂, σ₂²)) = ln(σ₂/σ₁) + (σ₁² + (μ₁ − μ₂)²) / (2σ₂²) − 1/2. It grows with the squared distance between the means and with any mismatch in spread. Multivariate Gaussians have a similar formula involving the covariance matrices.
How is KL divergence used in a VAE?
The VAE loss is a reconstruction term plus KL(q(z|x) ‖ p(z)), where q is the encoder's Gaussian over latent codes and p(z) is a standard normal prior. For a standard normal prior the KL term reduces to 1/2 (μ² + σ² − 1 − ln σ²) per latent dimension. It keeps the latent space smooth, so sampling z from the prior produces sensible outputs.
Why does RLHF use a KL penalty?
During RLHF, the policy is rewarded by a learned reward model, which it can exploit by producing odd text that scores well. Subtracting β × KL(policy ‖ reference model) from the reward keeps the fine-tuned model close to the original. The coefficient β trades reward against staying close.
Related explainers
- Entropy & Cross-Entropy — Measuring surprise, uncertainty, and prediction quality
- Probability & Distributions — Spam, densities, and the math of uncertainty
- Loss Functions — How networks measure their mistakes
- Autoencoders & VAEs — From compression to generation — standard and variational autoencoders
- RLHF — Align language models with human preferences using reward models and policy optimization
- GANs — Two networks competing to create
References
- Kullback & Leibler (1951), "On Information and Sufficiency", Annals of Mathematical Statistics
- Kingma & Welling (2013), "Auto-Encoding Variational Bayes"
- Ouyang et al. (2022), "Training language models to follow instructions with human feedback"
- Szegedy et al. (2016), "Rethinking the Inception Architecture for Computer Vision" (label smoothing)
- Bishop (2006), "Pattern Recognition and Machine Learning", sections 1.6 and 10.1
Last updated