ENID
Lensa ML

How Probability Works

Step 1 of 6: What Is Probability?

Read emails and watch the spam share settle

00.250.50.7511101001000emails read (log scale)true share 0.30press +1 to read the first email
True spam share0.30
Emails read
0
Spam share so far
—
Off by
—
Band (±2σ)
—

Each email is spam with probability 0.3, but no single email tells you that. Read some and watch the share of spam you have seen. The shaded band is where 95% of runs end up.

Read emails one at a time, then in batches, and add new runs. The frequentist reading of probability is exactly this long-run share. The Bayesian reading treats 0.3 as a degree of belief before you read anything, and both follow the same rules. The band is ±2 standard errors, 2√(p(1 − p)/n), so the typical error shrinks like 1/√n: reading 4× as many emails only halves it. The gambler's fallacy is the belief that after a streak of spam the next email is more likely to be normal. Each email here is independent, so a streak changes nothing.

Step 2 of 6: A Distribution

How a total of 1 is shared across outcomes

simulate an inbox of 10 emails012345678910spam emails in an inbox of 10
Spam share p0.30
P(exactly 3 spam)
0.267
Simulated share
—
Inboxes simulated
0
All bars add to
1.000

A distribution shares out a total probability of 1 across every possible outcome. Here the outcomes are 0 to 10 spam emails, and the bars come from the binomial formula C(10, k)·pk·(1−p)10−k. The most likely count is 3, but it only happens 27% of the time. Simulate a few hundred inboxes and compare the white lines with the bars.

Simulate inboxes and compare the white lines with the bars. Each email is a Bernoulli trial, spam with probability p, and the number of spam emails in 10 independent trials follows a binomial distribution with mean 10p and variance 10p(1 − p). A binary classifier with a sigmoid output is predicting the p of one Bernoulli trial. Computers draw samples like these by comparing uniform random numbers with p, which is exactly what this demo does.

Step 3 of 6: Density Is Not Probability

For continuous values, probability is area

00.00660.01368.3%0100200300email length (words)density per word
From120
To180
Spread σ (words)30
P(length in range) = area
68.3%
Peak height
0.013
P(exactly 150)
0
Total area
1

Email length is continuous, so the chance of one exact length, exactly 150 words and not 150.01, is 0. Probability lives in ranges: 68.3% of emails are between 120 and 180 words, the shaded area. The curve's height, 0.013 at the peak, is a density in "probability per word". Switch to thousands of words to see why that height is not a probability.

Move the range handles, then switch units. For a continuous variable the curve is a density f(x), and P(a < X < b) is the area between a and b. The area is the same in any unit, but the height scales with the unit, so a density can be larger than 1. The curve here is a normal distribution with a mean of 150 words; a larger σ widens it and lowers the peak, because the total area must stay 1. Real email lengths are skewed and never negative, so a normal curve is only an approximation.

Step 4 of 6: Conditional Probability

Conditioning shrinks the world, and order matters

each area = share of all emailsspam + "free"spamnormal mailspam 30%normal 70%
P(spam)0.30
P("free" | spam)0.60
P("free" | normal)0.05
P("free")
0.215
P("free" | spam)
0.60
P(spam | "free")
0.837

Every email lands in one of four boxes, and each box's area is its share. P("free" | spam) = 0.60 asks what fraction of the red column is the bright top. P(spam | "free") asks the opposite: what fraction of the bright strips is red. Switch to "only emails with free" to see that second question.

Drag the three sliders and switch between the two views. Every area in the square is a probability, so the bright red box is P(spam and "free") = P(spam) × P("free" | spam). Conditioning keeps only the bright strips and rescales them, which is Bayes' theorem: P(spam | "free") = P("free" | spam) P(spam) / P("free"). Lower P(spam) to 0.02 and the same clue becomes much weaker evidence; this base-rate fallacy is behind many misread medical tests. The two events are independent exactly when both columns have the same "free" share.

Step 5 of 6: Expected Value

The point where a distribution balances

012345678910
Fulcrum5.0
Spam share p0.30
Fulcrum
5.0
Net pull Σ p(k)·(k − fulcrum)
-2.00
Balanced?
tips left

Find the point where the distribution balances. The beam tips left because the probability-weighted distances to the left outweigh the rest by 2.00. Slide the fulcrum until the pull is 0, then reveal E[X] to check. Try changing p first and predicting where it will balance.

Slide the fulcrum until the beam balances, then reveal E[X]. The expected value is the probability-weighted average Σ k·p(k), the point where the distribution balances, and it does not have to be a value X can take. Expectation is linear, so E[aX + bY] = aE[X] + bE[Y] even when X and Y are dependent; the binomial mean 10p is just ten emails each contributing p. Training a model means minimizing its expected loss over the data distribution, which in practice is estimated by the average loss on a sample.

Step 6 of 6: Learning Probabilities from Data

Why rare cases give shaky estimates

estimate of P(spam | word) from 1,000 training emails"free"232 emails contain it, 197 spam00.51"unclaimed"2 emails contain it, 1 spam00.51dashed = true value · dot = estimate · band = ±2σ
Training emails1k
Emails with "free"
232
Emails with "unclaimed"
2
Rare-word error
0.31

What matters is not the 1,000 training emails but how many contain the word: 232 for "free" and only 2 for "unclaimed". The rare word's estimate is 0.50 against a true 0.81, with a band of ±0.71. Draw new training samples and watch the rare word jump while the common one barely moves. This is the 1/√n law from step 1, applied inside each condition.

Change the training-set size and draw new samples. Naive Bayes and similar models estimate probabilities such as P(spam | word) by counting, and the error of each estimate depends on how many emails meet the condition, not on the total. Rare words, rare classes, and rare feature combinations get unreliable estimates even in a large dataset. Add-one (Laplace) smoothing adds a pseudo-count to every count, which amounts to using a prior. The same 1/√n law governs the noise in mini-batch gradients.

What are probability and probability distributions?

Probability measures how likely an event is, on a scale from 0 (impossible) to 1 (certain). A probability distribution assigns probabilities to every possible outcome of a random variable. For a discrete variable, such as a coin flip, it gives a probability for each outcome. For a continuous variable, such as a height, it gives a density curve, and probabilities are areas under that curve.

Machine learning is built on these ideas. Training data is treated as samples from an unknown distribution, classifiers output a distribution over classes, and the most common loss functions are negative log-probabilities.

Key ideas

  • Probabilities run from 0 to 1, and the probabilities of all outcomes sum to 1.
  • For continuous variables, probability is area under a density curve.
  • μ sets where the normal distribution sits; σ sets how wide it is.
  • P(B|A) and P(A|B) are different quantities; Bayes' theorem converts one into the other.
  • Expected value is a probability-weighted average, and it is linear.
  • Sample averages approach the true mean as n grows, with error shrinking like 1/√n.

Common questions

What is a random variable?

A random variable assigns a number to each outcome of a random process. For a coin flip, X might be 1 for heads and 0 for tails. Working with numbers instead of outcomes is what lets you compute averages, variances, and distributions.

What is the difference between probability and likelihood?

Probability fixes the model's parameters and asks how likely different outcomes are. Likelihood fixes the observed data and asks how well different parameter values explain it. Likelihood is not a distribution over parameters and does not have to sum to 1. Maximum likelihood estimation picks the parameters that make the observed data most likely.

Why do ML models work with log-probabilities?

The probability of a whole dataset is a product of many small numbers, which quickly underflows to zero on a computer. Taking logs turns the product into a sum, and because log is increasing, the parameters that maximize the log-likelihood also maximize the likelihood. Cross-entropy loss is the negative log-probability of the correct answer.

Which distributions show up most in machine learning?

Bernoulli and categorical distributions describe classification outputs. The normal distribution models regression noise, weight initialization, and the latent space of VAEs. Uniform distributions are used for initialization and random sampling, and the Poisson distribution models counts.

Related explainers

References

  1. Blitzstein & Hwang, "Introduction to Probability" (free online edition)
  2. Goodfellow, Bengio & Courville (2016), "Deep Learning", chapter 3: Probability and Information Theory
  3. Bishop (2006), "Pattern Recognition and Machine Learning", chapters 1–2

Last updated