How KL Divergence Works
Two Distributions
Compare P and Q visually
Both P and Q are nearly uniform — high entropy means maximum uncertainty in both.
Step 1 of 6: Two Distributions
Compare P and Q visually
Both P and Q are nearly uniform — high entropy means maximum uncertainty in both.
Step 2 of 6: The KL Formula
Information gap, term by term
There is a moderate information gap between P and Q.
Step 3 of 6: Asymmetry
KL(P‖Q) ≠ KL(Q‖P)
The two directions are nearly equal here — but that's a special case, not the norm!
Step 4 of 6: KL = Cross-Entropy − Entropy
Breaking down the divergence
Cross-entropy is always ≥ entropy. The extra cost is exactly the KL divergence — the price of using Q instead of P.
Step 5 of 6: Mode-Seeking vs Mode-Covering
How the direction changes fitting behavior
Mode-covering: KL(P‖Q) penalizes Q for missing anywhere P has mass. Q must spread to cover both peaks — try increasing spread!
Step 6 of 6: Cross-Entropy Loss
How classifiers learn from KL
The model is unsure — Cat isn't even the top prediction. High cross-entropy loss means the model would receive a strong gradient update.