1Probability & Distributions 2Statistics Essentials 3Bayes' Theorem 4Hypothesis Testing 5Sampling & Monte Carlo 6MLE & MAP Estimation 7Entropy & Cross-Entropy 8KL Divergence

Lensa ML

How MLE & MAP Work

Step 1 of 6: Likelihood of Data

How probable is the data given a parameter?

LIKELIHOOD FORMULA

L(p) = p^k · (1 − p)^N−k

= 0.50⁵ · 0.50⁵ = 9.77e-4

p (prob. of heads)0.50

N (number of flips)10

k (heads observed)5

N flips

k heads

0.50

L(p)

9.77e-4

Your p = 0.50 is very close to the observed ratio k/N = 0.50. The likelihood is near its maximum, so this p makes the observed data about as probable as it can be.

Drag p, N and k. The formula L(p) = p^k (1 − p)^(N−k) multiplies one factor per flip, which assumes the flips are independent and identically distributed. It depends only on the count k, not on the order of heads and tails, so k is a sufficient statistic: nothing else in the data helps estimate p. The binomial coefficient C(N, k) is left out because it does not depend on p. At the defaults (p = 0.5, N = 10, k = 5), L is 0.5^10 ≈ 9.8 × 10⁻⁴.

Step 2 of 6: Log-Likelihood

Taking logs for numerical stability

LOG-LIKELIHOOD FORMULA

log L(p) = k·log(p) + (N−k)·log(1−p)

= 5·log(0.50) + 5·log(0.50) = -6.93

p (prob. of heads)0.50

N (number of flips)10

k (heads observed)5

log L(p)

-6.93

L(p)

9.77e-4

k / N

0.50

The log-likelihood is at its peak here. The peak sits at the same p as the raw likelihood's, because log never changes which value is largest.

Drag p, N and k again and compare this curve with the previous step. For the Bernoulli model, log L(p) is concave, so it has exactly one peak and gradient ascent cannot get stuck. Its curvature at the peak is −N/(p̂(1 − p̂)), which is −40 at the defaults. A sharper peak means the data pins p down more tightly. This curvature is called the Fisher information, and its inverse gives the usual standard error of the estimate. Neural network log-likelihoods are not concave, so training can end in many different optima.

Step 3 of 6: Maximum Likelihood Estimation

Find the parameter that makes data most probable

FINDING THE MLE: SET DERIVATIVE TO ZERO

log L(p) = k·log(p) + (N−k)·log(1−p)

d/dp log L = k/p − (N−k)/(1−p) = 0

k·(1−p) = (N−k)·p → k = N·p

p̂ = k / N = 6 / 12 = 0.5000

N (number of flips)12

k (heads observed)6

Your guess p0.50

MLE (k/N)

0.50

Your p

0.50

log L(MLE)

-8.32

log L(your p)

-8.32

Your guess matches the MLE. The derivation above shows why: setting the derivative of log L to zero gives p̂ = k/N = 6/12 = 0.50. No other value can produce a higher likelihood.

Drag N, k and Your guess p, and compare your log-likelihood with the green MLE marker. Setting the derivative to zero works for this model because the log-likelihood is smooth and concave. The same recipe gives other familiar estimators: the MLE of a Gaussian's mean is the sample average, and the MLE of its variance divides by N. The MLE is invariant to reparameterization. If p̂ = 0.6, the MLE of the odds p/(1 − p) is 1.5. For neural networks there is no closed form, so the likelihood is maximized numerically by gradient descent.

Step 4 of 6: The Problem with MLE

Small samples lead to extreme estimates

Sample size N5

True p

0.50

MLE (sample 1)

0.20

MLE spread

0.80

With N = 5, the MLE estimates still vary considerably (spread = 0.80). Some estimates are far from the true p = 0.5. More data helps, and so does adding prior knowledge.

Drag Sample size N from 1 to 50. Each dot is the MLE from one of 20 simulated experiments with a fair coin. With N = 3, an experiment with no heads gives p̂ = 0, which says heads is impossible. The same problem appears in language models and Naive Bayes, where a word missing from the training data gets zero probability. A standard fix is Laplace smoothing, (k + 1)/(N + 2), which Laplace derived as the posterior mean under a uniform prior. Adding a prior gives the same kind of correction.

Step 5 of 6: Adding a Prior (MAP)

Beliefs plus evidence yield better estimates

likelihoodpriorposterior

N (number of flips)10

k (heads observed)5

Prior strength α (Beta(α,α))5

Prior mode

0.50

MLE (k/N)

0.50

MAP estimate

0.50

With α = 5, the prior pulls the MAP estimate (0.50) toward 0.5, away from the MLE (0.50). The posterior is a compromise between data and prior belief.

Drag N, k and Prior strength α. A Beta(α, α) prior acts like 2α − 2 imaginary flips, half heads and half tails, which is why MAP = (k + α − 1)/(N + 2α − 2). Try N = 10, k = 9 and α = 5: the MLE is 0.90 and MAP is 13/18 ≈ 0.72. MAP is the posterior's mode. The posterior mean, (k + α)/(N + 2α), is 0.70 here. For a skewed posterior the two differ, and unlike the MLE, the MAP estimate changes if you reparameterize.

Step 6 of 6: MLE vs MAP

MAP estimation is MLE with a regularization penalty

log L (MLE objective)log prior (penalty)log L + log prior (MAP objective)

Prior strength α5

N (number of flips)10

k (heads observed)5

MLE (k/N)

0.500

MAP

0.500

Difference

0.000

log prior at MLE (unnormalised)

-5.5

The prior (α = 5) and data (N = 10) are balanced. MAP = 0.500 sits between MLE = 0.500 and the prior mode 0.5. The penalty prevents extreme estimates without ignoring the data.

Drag Prior strength α, N and k, and compare the blue, green and orange curves. Taking logs turns MAP into log-likelihood plus log-prior, so the prior becomes an additive penalty. Here the penalty is (α − 1)[log p + log(1 − p)], which is largest at p = 0.5. For neural network weights, a Gaussian prior N(0, σ²) gives the penalty −‖w‖²/(2σ²), which is L2 regularization with λ = 1/(2σ²). A Laplace prior gives an L1 penalty, as in the lasso. The penalty's weight stays fixed while the log-likelihood grows with N.

What are MLE and MAP estimation?

Maximum likelihood estimation (MLE) picks the parameter values that make the observed data most probable. Maximum a posteriori (MAP) estimation multiplies the likelihood by a prior over the parameters first, so it picks the most probable parameters given the data. With a flat prior the two give the same answer. With little data, the prior pulls the MAP estimate toward values that were plausible beforehand.

Most model training is one of the two. Minimizing mean squared error is MLE under Gaussian noise, and minimizing cross-entropy is MLE for a classifier's predicted probabilities. Adding an L2 penalty or weight decay turns that into MAP with a Gaussian prior on the weights, and an L1 penalty corresponds to a Laplace prior.

Key ideas

The likelihood is the probability of the observed data viewed as a function of the parameters.
MLE picks the parameters that maximize the likelihood; for coin flips it gives p̂ = k/N.
Maximizing the log-likelihood gives the same answer as maximizing the likelihood.
With small samples, MLE can give extreme estimates such as p̂ = 0.
MAP maximizes likelihood × prior, so the prior acts as a regularizer.
A Gaussian prior on weights gives L2 regularization, and a Laplace prior gives L1.

Common questions

What is the difference between MAP and full Bayesian inference?

MAP keeps a single point, the peak of the posterior. Full Bayesian inference keeps the whole posterior and averages predictions over it, which also gives a measure of uncertainty. MAP is much cheaper, but it ignores how wide the posterior is.

Is minimizing mean squared error the same as maximum likelihood?

Yes, when the targets are the model's output plus Gaussian noise with a fixed variance. The negative log-likelihood is then Σ(y − f(x))² / (2σ²) plus a constant, so minimizing it is the same as minimizing squared error. With heavy-tailed noise, MLE gives a different loss, such as absolute error for Laplace noise.

Is the maximum likelihood estimate always unbiased?

No. The MLE of a Gaussian's variance divides by N and is biased low, which is why the sample variance divides by N − 1. MLE is consistent, though: under standard conditions it converges to the true parameter as N grows.

What is the EM algorithm?

Expectation-maximization finds MLE or MAP estimates when the model has hidden variables, such as which cluster produced each point in a Gaussian mixture. It alternates between estimating the hidden variables from the current parameters and re-fitting the parameters to those estimates. Each round never lowers the likelihood, though it can stop at a local maximum.

Related explainers

Probability & Distributions — Spam, densities, and the math of uncertainty
Bayes' Theorem — Updating beliefs with evidence
Regularization — Dropout, L1, and L2 — keeping models honest
Loss Functions — How networks measure their mistakes
Naive Bayes Classifier — Probabilistic classification with Bayes' rule
Statistics Essentials — Mean, variance, standard deviation, and the Central Limit Theorem

References

Last updated October 25, 2026