1Probability & Distributions 2Statistics Essentials 3Bayes' Theorem 4Hypothesis Testing 5Sampling & Monte Carlo 6MLE & MAP Estimation 7Entropy & Cross-Entropy 8KL Divergence

Lensa ML

How Entropy & Cross-Entropy Work

Step 1 of 6: Surprise

Unlikely events carry more information

Rare (0.01)Certain (1.0)

P(event)

0.50

Surprise

1.00 bits

Likely events carry at most 1 bit. You mostly expected them already.

Drag the slider from Rare (0.01) to Certain (1.0). Surprise, also called self-information, is I(x) = −log₂ P(x), measured in bits. The log makes information add up: two independent events have probability P(a)·P(b), and since log turns products into sums, their surprises add. Each halving of probability adds exactly one bit, so P = 0.5 gives 1 bit and P = 0.25 gives 2. Using the natural log instead gives units called nats, and 1 nat is about 1.44 bits.

Step 2 of 6: Entropy

The average surprise of a distribution

P(X) = 0P(X) = 1

H(X) = −Σ P(X)·log₂(P(X))

= −[0.50×log₂(0.50) + 0.50×log₂(0.50)]

= −[-0.500 + -0.500] = 1.000 bits

P(X)

0.50

1 − P(X)

0.50

Entropy H(X)

1.000 bits

Maximum entropy. With both outcomes equally likely, uncertainty peaks at 1 full bit.

Drag P(X) from 0.01 to 0.99. The curve is the binary entropy function H(p) = −p log₂ p − (1 − p) log₂(1 − p). It is symmetric around 0.5 because swapping the labels of the two outcomes changes nothing. Entropy is also a limit on compression. Shannon's source coding theorem says no lossless code can use fewer than H(X) bits per symbol on average. A coin with p = 0.9 has an entropy of about 0.47 bits, so a long record of its flips compresses to roughly half its raw length.

Step 3 of 6: Distributions & Entropy

Shape determines uncertainty

A: 25

B: 25

C: 25

D: 25

Entropy H

2.000 bits

Max H

2.000 bits

Near-uniform distribution: entropy is close to the 2-bit maximum for 4 outcomes.

Drag the four sliders for A, B, C and D. The values are weights, normalized so the bars always sum to 100%. With n outcomes, entropy runs from 0, when one outcome takes all the probability, up to log₂ n when all are equal: 2 bits here, since 4 = 2². The quantity 2^H reads as the effective number of equally likely outcomes. Language models report the same idea, computed from cross-entropy, as perplexity. A perplexity of 20 means the model is as uncertain as a uniform choice among 20 tokens.

Step 4 of 6: Cross-Entropy

Measuring the mismatch between distributions

Adjust predicted distribution (q):

Cat q

Dog q

Bird q

H(p)

1.157 bits

H(p,q)

1.446 bits

KL gap

0.290 bits

A fair match. The KL gap is small: q costs a fraction of a bit more per sample than p would.

Drag the Cat q, Dog q and Bird q sliders while the true distribution p stays at 70%, 20% and 10%. Cross-entropy is H(p, q) = −Σ p(x) log₂ q(x), and it splits as H(p, q) = H(p) + KL(p‖q). With the starting q of 40/30/30, H(p) is 1.157 bits, H(p, q) is 1.446 bits, and the KL gap is 0.290 bits. Cross-entropy is not symmetric. Swapping p and q gives a different number, and only p, the true distribution, sits outside the log.

Step 5 of 6: Cross-Entropy as Loss

The ML connection — why this is the loss function

Wrong (0.01)Correct (1.0)

0.50

Loss

0.693 nats

∂L/∂ŷ

-2.0

The model leans wrong. Loss is above 0.69 nats and the gradient pushes harder to raise ŷ.

Drag ŷ from Wrong (0.01) to Correct (0.99). The true label is fixed at the positive class, so cross-entropy reduces to L = −log ŷ. The log here is natural, so the loss is in nats: 0.693 at ŷ = 0.5 and 4.6 at ŷ = 0.01. The gradient card shows ∂L/∂ŷ = −1/ŷ. In a real model ŷ comes from a sigmoid or softmax, and the chain rule through it gives a gradient on the logit of just ŷ − y. Most frameworks take raw logits and fuse the two steps for numerical stability.

Step 6 of 6: Why Minimize Cross-Entropy

Watch training converge to the true distribution

Epoch

H(p,q)

1.589 bits

KL(p‖q)

0.236 bits

Before training, the model predicts a uniform q, so cross-entropy starts high. This run is illustrative: q is moved toward p on a smooth schedule, not by a real optimiser. Press Play or step through epochs.

Press Play, or use ← Step and Step → to move through the 60 epochs. The true distribution is 60/25/15, so the dashed baseline H(p) is 1.353 bits, and the near-uniform starting guess costs 1.589. The curve is a smooth simulation of q moving toward p, not a real training run. Because H(p) does not depend on the model, minimizing cross-entropy is the same as minimizing KL(p‖q). The KL(p‖q) card shows that gap directly: 0.236 bits at the start, 0.033 by epoch 15, and close to zero by the end.

What are entropy and cross-entropy?

Entropy is the average amount of surprise in a random outcome, measured in bits. It comes from information theory, the mathematics of measuring and transmitting information that Claude Shannon founded in 1948. Cross-entropy measures how many bits you need, on average, when data comes from one distribution but you encode it with a code built for another.

In machine learning, cross-entropy is the standard loss for classification, from logistic regression to the next-token loss of large language models. Related quantities appear across the field: KL divergence in VAEs and knowledge distillation, information gain in decision trees, perplexity for evaluating language models, and mutual information for feature selection.

Key ideas

An event with probability p carries −log₂ p bits of surprise.
Entropy is the expected surprise of a distribution.
Entropy is highest for a uniform distribution, at log₂ n bits for n outcomes.
Cross-entropy H(p, q) equals H(p) + KL(p‖q), so it is never below H(p).
Minimizing cross-entropy on labeled data is the same as maximizing the log-likelihood of the labels.
Bits use log base 2, and nats use the natural log.

Common questions

What is the difference between binary and categorical cross-entropy?

Binary cross-entropy is for two classes or for independent yes/no labels: L = −[y log ŷ + (1 − y) log(1 − ŷ)], with ŷ from a sigmoid. Categorical cross-entropy is for one label out of K classes: L = −log q(correct class), with q from a softmax. Binary cross-entropy is the K = 2 case of the categorical form.

Why use cross-entropy instead of mean squared error for classification?

With a sigmoid output, the MSE gradient includes the sigmoid's slope, which is close to zero when the output saturates. A confidently wrong prediction then learns very slowly. Cross-entropy cancels that factor, so the gradient stays large exactly when the model is badly wrong. It is also the negative log-likelihood of a Bernoulli or categorical model, which gives it a probabilistic meaning.

What is information gain in a decision tree?

Information gain is the drop in label entropy after a split: the entropy of the parent node minus the weighted average entropy of the child nodes. ID3 picks the split with the largest gain, and C4.5 uses a normalized version called the gain ratio. A split that sends every class to its own pure child removes all the entropy.

What is mutual information?

Mutual information I(X; Y) = H(X) − H(X|Y) is the reduction in uncertainty about X after you learn Y. It is zero exactly when X and Y are independent, and unlike correlation it also detects nonlinear dependence. It is used for feature selection and in self-supervised learning objectives.

Related explainers

KL Divergence — Measuring how different two distributions are
Probability & Distributions — Spam, densities, and the math of uncertainty
Loss Functions — How networks measure their mistakes
Softmax, Logits & Temperature — Turning raw scores into probabilities
Decision Trees & Splitting — Recursive partitioning, Gini impurity, and pruning
Language Modeling — Predict the next token — the objective that powers GPT, BERT, and modern NLP

References

Last updated October 25, 2026