1Probability & Distributions 2Statistics Essentials 3Bayes' Theorem 4Hypothesis Testing 5Sampling & Monte Carlo 6MLE & MAP Estimation 7Entropy & Cross-Entropy 8KL Divergence

Lensa ML

Hypothesis Testing

Step 1 of 6: Null vs Alternative

Two competing explanations for the data

True effect size (μ₁)1.50

Effect size

1.50

H₀ mean

0.00

H₁ mean

1.50

Separation

Medium

The distributions are separating. H₀ says the data comes from μ=0; H₁ says μ=1.5. The shaded overlap is where either hypothesis could easily have produced the data.

Drag True effect size (μ₁) from 0 to 3.5. Both curves have σ = 1, so the effect is measured in standard deviations, the same scale as Cohen's d. Cohen called d = 0.2 small, 0.5 medium and 0.8 large. A test never proves H₁. It asks whether the data would be surprising under H₀, which is why H₀ is the simple, specific claim (μ = 0) and H₁ covers everything else. In ML, H₀ is usually "the new model is no better than the baseline," with both evaluated on the same test set.

Step 2 of 6: Test Statistics

Measuring how extreme the observed data is under H₀

Observed sample mean1.20

z-statistic

1.20

p-value

0.2301

Tail area

23.01%

z = 1.20. The pink tails hold the probability, under H₀, of a value at least this far from 0 in either direction: p = 0.230.

Drag Observed sample mean from −3 to 3. Here σ = 1 and n = 1, so z equals the observed value. In general the z statistic is z = (x̄ − μ₀)/(σ/√n), the distance from the null value measured in standard errors. When σ is unknown and estimated from the sample, the same ratio follows a t-distribution with n − 1 degrees of freedom. Other tests use other statistics, such as χ² for counts in a table or F for comparing several group means, but each one is compared against its distribution under H₀.

Step 3 of 6: P-Values

The probability of seeing data this extreme if H₀ is true

Observed value1.80

Observed

1.80

p-value

0.0719

-log₁₀(p)

1.14

p = 0.072. The tails are narrowing, but under H₀ data this extreme would still appear 7.2% of the time. Suggestive, not conclusive.

Drag Observed value from 0 to 3.5. At the default of 1.8, the two-tailed p-value is 0.072. A p-value is P(data at least this extreme | H₀). It is not the probability that H₀ is true, and 1 − p is not the probability that the effect is real; those need a prior, as in Bayes' theorem. A one-tailed test counts only one side and halves p, but the direction must be chosen before seeing the data. When H₀ is true, p-values are uniformly distributed between 0 and 1.

Step 4 of 6: Significance Level α

The threshold for rejecting H₀ — you choose it before looking at data

Significance level α0.05

0.050

z-critical

±1.96

p-value

0.0357

Decision

Reject

At α = 0.050, the critical value is ±1.96. The observed z = 2.1 falls in the pink rejection region (p = 0.0357 < α), so H₀ is rejected.

Drag Significance level α from 0.01 to 0.10 while the observed z stays at 2.1, which has p = 0.036. The critical value is the z where each tail holds α/2: 1.96 for α = 0.05 and 2.58 for α = 0.01. Rejecting when p < α and rejecting when |z| > z_crit are the same rule. The 0.05 convention goes back to Ronald Fisher in the 1920s. Fields that run many tests use stricter levels: particle physics requires 5σ, about p = 3 × 10⁻⁷, before announcing a discovery.

Step 5 of 6: Type I & Type II Errors

False positives vs false negatives — you can't minimize both

Significance level α0.05

True effect size1.50

Type I (α)

0.050

Type II (β)

0.677

Power (1−β)

0.323

Low power (32%). The alternative (purple) overlaps heavily with the acceptance region. β = 0.68: the test misses the effect most of the time. A larger effect or a higher α raises power.

Drag Significance level α and True effect size. A Type I error rejects H₀ when it is true, with probability α. A Type II error misses a real effect, with probability β, and power is 1 − β. At the defaults (α = 0.05, effect 1.5), power is only about 32%, because a single observation is not much evidence for an effect of 1.5σ. In classification terms, α is the false positive rate and power is recall. Lowering α trades one error for the other, like moving a classifier's decision threshold.

Step 6 of 6: Power & Sample Size

How many samples you need to reliably detect an effect

Sample size n20

Power

60.9%

Sample n

n for 80%

0.224

Power = 61%, better but below the conventional 80% target. The standard error shrinks as 1/√n, so each sample helps. 80% needs n ≈ 32.

Drag Sample size n from 5 to 200 along the power curve. The effect is fixed at d = 0.5 with α = 0.05, and at n = 20 power is about 61%. The required sample size is n = ((z_α/2 + z_β)/d)², which gives ((1.96 + 0.84)/0.5)² ≈ 31.4, rounded up to 32. Halving the effect size quadruples the required n. This is why comparing two close ML models needs a large test set: on 1,000 examples, a 0.5% accuracy gap is well inside the noise.

What is hypothesis testing?

Hypothesis testing is a procedure for deciding whether data give enough evidence to reject a default claim, called the null hypothesis H₀. You compute a test statistic from the data, find how likely a value at least that extreme would be if H₀ were true (the p-value), and reject H₀ if that probability falls below a threshold α chosen in advance.

In machine learning, hypothesis tests check whether a new model beats a baseline or just got lucky on the test set, using tools such as McNemar's test or a paired bootstrap. They also decide A/B tests of deployed models, detect drift between training and production data, and flag features whose link to the label could be chance.

Key ideas

H₀ is the default claim of no effect, and a test asks whether the data are surprising under it.
A test statistic measures how far the data sit from H₀, in units of standard error.
The p-value is the probability, assuming H₀ is true, of data at least as extreme as what was observed.
α is the false positive rate you accept, and it is fixed before looking at the data.
Power is the probability of detecting an effect that is really there.
The required sample size grows with the inverse square of the effect size.

Common questions

What is the multiple comparisons problem?

Each test at α = 0.05 has a 5% false positive rate, so running 20 independent tests on pure noise gives at least one "significant" result about 64% of the time. The Bonferroni correction tests each at α/m for m tests. The Benjamini-Hochberg procedure controls the false discovery rate instead, which keeps more power when there are many tests.

What is the difference between statistical and practical significance?

Statistical significance says an effect is unlikely to be zero. Practical significance says it is large enough to matter. With millions of samples, a 0.01% improvement can have a tiny p-value and still be worthless, so report effect sizes and confidence intervals along with p-values.

How do you test whether one ML model is better than another?

When both models are scored on the same test set, use a paired test. McNemar's test compares the counts of examples that only one of the two models gets right. A paired bootstrap resamples the test set many times and checks how often the accuracy gap changes sign. Dietterich (1998) also recommends the 5×2 cross-validation paired t-test.

Does failing to reject H₀ mean H₀ is true?

No. It means the data were not strong enough to rule H₀ out. With low power, a real effect is easy to miss, so a non-significant result from a small sample says very little. Equivalence tests are the right tool for showing that an effect is close to zero.

Related explainers

Statistics Essentials — Mean, variance, standard deviation, and the Central Limit Theorem
Probability & Distributions — Spam, densities, and the math of uncertainty
Bayes' Theorem — Updating beliefs with evidence
Model Evaluation & Metrics — Precision, recall, F1, ROC curves, and regression metrics
Cross-Validation — K-fold, stratified, and leave-one-out evaluation
Sampling & Monte Carlo — Random sampling, the law of large numbers, and Monte Carlo estimation

References

Last updated October 25, 2026