ENID
Lensa ML

How Statistics Works

Step 1 of 6: Mean — The Center of Gravity

Where the balance point of a data set lies

Number of data points7
012345678910μ = 3.63The mean is the "balance point" — it minimizes squared distances
N points
7
Mean
3.63
Sum
25.4

The points are moderately spread out. The mean still balances them, with points on each side pulling against each other.

Drag Number of data points from 3 to 15 and watch the triangle under the line. The mean x̄ = (x₁ + … + xₙ)/n is the point where the deviations sum to exactly zero, and it is the value c that minimizes Σ(xᵢ − c)². That squared-error property is why a regression trained with MSE loss predicts a conditional mean. The median minimizes Σ|xᵢ − c| instead, so one extreme value can drag the mean a long way while barely moving the median. Skewed data such as incomes is the usual example.

Step 2 of 6: Variance & Standard Deviation

How spread out data is around the mean

σ (standard deviation)1.50
-σ+σμ=0~68% of data falls within ±1σ of the mean
Variance (σ²)
1.96
Std dev (σ)
1.40
Range
6.3

Moderate spread. The data fans out, but most values still fall within 2 standard deviations of the center.

Drag σ from 0.5 to 4.0. Variance is the average squared deviation from the mean, σ² = Σ(xᵢ − x̄)²/n, and the standard deviation σ is its square root, which puts spread back into the data's own units. The readouts come from 40 points drawn with the slider's σ, so they land near the chosen value but not exactly on it. For a normal distribution, about 68% of values fall within ±1σ, 95% within ±2σ and 99.7% within ±3σ. Standardizing a feature means subtracting its mean and dividing by its σ.

Step 3 of 6: Populations vs Samples

Why samples are noisy estimates of the truth

Sample size (n)30
1030507090pop μx̄ = 47.4PopulationSample
Pop. mean
50.2
Sample mean
47.4
Error
2.79
Sample size
30

At n = 30, the sample histogram is starting to resemble the population shape. The sample mean (x̄ = 47.4) is getting closer to the true mean.

Drag Sample size (n) from 5 to 200. The population is 500 values drawn from a normal distribution with mean 50 and σ = 12, and the sample is its first n values. The sample mean x̄ estimates the population mean μ, and it is unbiased: averaged over many samples, it equals μ. For the sample variance, dividing by n − 1 instead of n (Bessel's correction) corrects for deviations from x̄ running slightly smaller than deviations from μ. In ML, the training set is the sample and future inputs are the population.

Step 4 of 6: The Central Limit Theorem

Sample means approach a bell curve as n grows

Sample size (n)5
13579Distribution of sample means (500 experiments)Sample meansNormal fit
Source dist.
Uniform
Sample size
5
Mean of means
5.01
Std error
1.29

The bell shape is emerging. At n = 5, the distribution of means is visibly concentrating around 5 with std error ≈ 1.29.

Drag Sample size (n) from 2 to 50. The histogram counts the means of 500 experiments, and every value comes from a uniform distribution on [0, 10] with mean 5 and σ = √(100/12) ≈ 2.89. The Std Error card is σ/√n: about 1.29 at n = 5 and 0.41 at n = 50. The theorem needs independent draws and a finite variance. Heavy-tailed sources converge slowly, and the Cauchy distribution, which has no finite variance, never converges: the mean of n Cauchy values is as spread out as a single value.

Step 5 of 6: Confidence Intervals

Quantifying uncertainty in estimates

Confidence level0.95
Sample size (n)50
spread of x̄ around μ-0.520.04μ95% CI around x̄ = -0.24 contains μ
Conf. level
95%
z-score
1.96
Margin
±0.277
Sample mean
-0.240

The 95% level is close to the usual 95% choice. It balances width against reliability, with a z-score of 1.96. This sample's interval contains μ.

Drag the Confidence level from 80% to 99% and Sample size (n) from 10 to 200. The interval is x̄ ± z·σ/√n, where z is the normal quantile for the chosen level, such as 1.96 for 95%. Here σ = 1 is known, the interval is built around the mean x̄ of one fixed sample, and the curve shows how x̄ spreads around the true mean μ. With n = 50, x̄ = −0.24 and the 95% margin is ±0.277, so the interval contains μ. At 90% the margin shrinks to ±0.233 and the interval just misses. Quadrupling n halves the width, and raising the level from 95% to 99% widens it by about 31%. When σ must be estimated from the sample, a slightly larger t-value replaces z.

Step 6 of 6: Why Statistics Matters for ML

Batch statistics and mini-batch noise

Batch size32
Batch meansμBatch varianceBatch index (20 mini-batches)
Batch size
32
Batch mean
0.590
Batch var
0.949
SD of batch means
0.134

Moderate batch size. The batch means vary with SD ≈ 0.134, close to the theory value σ/√n = 0.141, so gradient estimates are reasonable. Sizes in this range are common defaults.

Drag Batch size from 4 to 128. The top chart shows the mean of each of 20 mini-batches drawn from activations with true mean 0.5 and σ = 0.8. The bottom chart shows each batch's variance against the true 0.64. At a batch size of 32, the batch means scatter with a standard deviation of about 0.8/√32 ≈ 0.14. Batch normalization standardizes each layer with exactly these noisy batch statistics during training, then switches to running averages at inference. Mini-batch gradients are batch means too, so their noise follows the same rule.

What statistics do you need for machine learning?

Statistics is the study of drawing conclusions from data under uncertainty. Descriptive statistics summarize a dataset with numbers such as the mean and the standard deviation. Inferential statistics use a sample to estimate properties of the larger population it came from, and to measure how uncertain those estimates are.

Machine learning relies on both. Feature scaling uses means and standard deviations, batch normalization recomputes them for every mini-batch, and stochastic gradient descent treats each mini-batch gradient as a sample estimate of the full gradient. Standard errors and confidence intervals tell you whether a 0.5% accuracy gain on a test set is a real improvement or noise.

Key ideas

  • The mean is the balance point of the data and minimizes the sum of squared deviations.
  • Variance is the average squared deviation, and the standard deviation is its square root.
  • A sample statistic estimates a population value and changes from one sample to the next.
  • The central limit theorem says sample means are approximately normal for large n, given finite variance.
  • The standard error of the mean is σ/√n.
  • A 95% confidence interval comes from a procedure that captures the population value in 95% of repeated samples.

Common questions

What is the difference between standard deviation and standard error?

The standard deviation describes how spread out individual data points are. The standard error describes how much an estimate, such as the sample mean, would vary across repeated samples. For the mean, SE = σ/√n, so it shrinks as the sample grows while the standard deviation of the data does not.

How large does n need to be for the central limit theorem?

A common rule of thumb is n ≥ 30, but the real answer depends on the source distribution. Symmetric distributions like the uniform look normal after about 5 to 10 values. Strongly skewed data, such as incomes or response times, can need hundreds.

What is the difference between a parameter and a statistic?

A parameter is a fixed property of the population, such as its true mean μ, and is usually unknown. A statistic is computed from a sample, such as the sample mean x̄, and is used to estimate the parameter. Greek letters usually denote parameters and Latin letters statistics.

Why do ML papers report results over several random seeds?

Training involves random initialization, data shuffling and dropout, so one run gives one sample of a random outcome. Reporting the mean and standard deviation across seeds shows how much of a difference between methods is noise. A gap smaller than the seed-to-seed spread is weak evidence that one method is better.

Related explainers

References

  1. James, Witten, Hastie & Tibshirani, "An Introduction to Statistical Learning" (free online edition)
  2. Ioffe & Szegedy (2015), "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift"
  3. Goyal et al. (2017), "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour"

Last updated