ENID
Lensa ML

Bias-Variance Tradeoff

Step 1 of 6: What is Bias?

Systematic underfitting

-2-112-11true: sin(x)model
Polynomial Degree2
Degree
2
Bias²
0.178
Train err
0.248

High bias: the curve is too simple to follow the sine wave. Degree 2 does no better than a line, because sin(x) is an odd function and an x² term cannot help.

Drag the polynomial degree from 1 to 10 and compare the fitted curve with the dashed sin(x). Bias is defined over many training sets, not one: it is the gap between the average prediction E[f̂(x)] and the target f(x), squared and averaged over x. A straight line has high bias on a sine wave no matter how many samples it gets, because no line can bend. The Train err card is measured on these 20 points, while the Bias² card is measured over 200 fresh training sets. Bias² is 0.178 at degrees 1 and 2, since an odd function like sin(x) gains nothing from an x² term, and falls to 0.004 at degree 3.

Step 2 of 6: What is Variance?

Sensitivity to training data

-2-112-11
Polynomial Degree2
Degree
2
Variance
0.012
Spread
0.065

Low variance: fits from five different noisy samples stay close.

Drag the degree and watch the five colored fits. Each is trained on a fresh draw of 20 noisy points from the same sin(x), which is what variance measures: E[(f̂(x) − E[f̂(x)])²], how far one model's prediction strays from the average model's prediction. The Spread card is the measured standard deviation across these five curves, averaged over x. Variance shrinks as the training set grows. For a least-squares fit it averages about σ²·p/n over the training inputs, where p is the number of coefficients and n the number of points.

Step 3 of 6: The U-Shaped Curve

Error = Bias² + Variance + Noise

Model ComplexityError14812Bias²VarianceTotal ErrorNoise Floor
Model Complexity4
Bias²
0.004
Variance
0.020
Noise
0.09
Total
0.114

Near the bottom of the U: bias² and variance together are close to their minimum.

Drag the model complexity from 1 to 12 and watch the four readouts. For squared error at a point x, expected test error splits exactly into three parts: E[(y − f̂(x))²] = Bias² + Variance + σ², where σ² is the noise in y itself. Here the data noise has variance σ² = 0.09, so no model can score below that floor. Bias² and variance are measured by fitting each degree on 200 fresh training sets. The minimum total, about 0.109, sits at degree 3, and degree 12 rises to 0.165 as variance grows. The split is exact only for squared error. For 0-1 classification loss, bias and variance interact, and several competing definitions exist.

Step 4 of 6: Underfitting Zone

High bias, low variance

-2-112-11Underfitting
Complexity1
Bias²
0.178
Variance
0.008
Total err
0.276
Diagnosis
Underfit

More data won't fix this. The model needs more flexibility. The red lines show how far each prediction misses.

Drag the complexity from 1 to 5 and watch the red residual lines shrink. In the high-bias zone, training error and test error are both high and close together, which is the signature to look for in practice. The remedies all add capacity: a higher degree, more input variables, a deeper network, or weaker regularization. Collecting more data does little, because bias does not depend on sample size. A degree-1 line fit to sin(x) on [−3, 3] still misses the curve's turns with a million points.

Step 5 of 6: Overfitting Zone

Low bias, high variance

-2-112-11Overfitting
Complexity12
Bias²
0.000
Variance
0.075
Total err
0.165
Diagnosis
Overfit

Extreme overfitting: the curve chases the noise. More data or regularization would reduce the variance.

Drag the complexity from 8 to 12. A degree-12 polynomial has 13 coefficients for 20 points, nearly enough freedom to pass through every one. Its bias is close to zero, yet each new sample would produce a very different curve, especially near x = ±3, where high-degree polynomials swing hardest (Runge's phenomenon). Regularization trades a little bias for a large cut in variance: ridge regression shrinks the coefficients toward zero, so the fit stops chasing single points. Averaging many high-variance models, as random forests do, reduces variance the same way.

Step 6 of 6: The Sweet Spot

Balancing complexity

Model ComplexityError14812Train ErrorTest Errorgapoptimal
Model Complexity3
Optimal
3
Test err
0.109
Train-test gap
0.033

Sweet spot: this complexity gives close to the lowest expected test error.

Drag the complexity across the full range and follow the white marker along the test curve. In real projects the bias² and variance curves are never visible, because the true function is unknown. Only training and validation error can be measured, so the optimum is found by cross-validation: fit each candidate on k − 1 folds, average the held-out error, and keep the lowest. Large neural networks can show a second descent past the point where they fit the training data exactly, with test error falling again as parameters keep growing.

What is the bias-variance tradeoff?

The bias-variance tradeoff describes how a model's expected test error splits into two competing sources. Bias is the error from a model being too rigid to represent the true pattern; variance is the error from a model being so flexible that its fit changes with every new training sample. Making a model more complex usually lowers bias and raises variance, so total error is lowest somewhere in between.

The idea guides most choices about model capacity: polynomial degree, tree depth, the number of neighbors in k-NN, the strength of L2 regularization, and the size of a neural network. It also explains why ensembles work. Bagging and random forests average many high-variance models to cut variance, while boosting adds weak, high-bias learners one at a time to cut bias.

Key ideas

  • Expected squared test error equals bias² plus variance plus irreducible noise.
  • Bias measures how far the average model is from the truth, across many training sets.
  • Variance measures how much the fitted model changes from one training set to the next.
  • More data lowers variance but leaves bias unchanged.
  • Regularization and averaging reduce variance at the cost of a small increase in bias.
  • The best complexity is chosen with validation data, because bias and variance cannot be measured directly.

Short takes on this topic

Common questions

Is high bias the same as underfitting?

They describe the same situation from two angles. High bias is the cause, a model class too rigid for the pattern, and underfitting is the symptom, high error on both the training and validation data. Likewise, high variance is the cause of overfitting.

Can a model have both high bias and high variance?

Yes. A model that is the wrong shape for the data and is also trained on very few, noisy examples can miss the pattern on average and still swing a lot between samples. A deep decision tree trained on inputs unrelated to the target is one example.

How does k in k-nearest neighbors affect bias and variance?

Small k gives low bias and high variance: with k = 1, each prediction copies a single neighbor, noise included. Large k averages more neighbors, which smooths the prediction and lowers variance, but it blurs real boundaries and raises bias. At k = n, every input gets the same prediction.

How do you estimate bias and variance in practice?

Variance can be estimated by training the same model on many bootstrap resamples and measuring how much the predictions spread at each test point. Bias needs the true function, so on real data it is approximated by subtracting that variance and a noise estimate from the measured test error. Simulations with a known function, like the sine wave here, give exact values.

Related explainers

References

  1. Geman, Bienenstock & Doursat (1992), "Neural Networks and the Bias/Variance Dilemma", Neural Computation
  2. Belkin, Hsu, Ma & Mandal (2019), "Reconciling modern machine learning practice and the bias-variance trade-off", PNAS
  3. Nakkiran et al. (2019), "Deep Double Descent: Where Bigger Models and More Data Hurt"
  4. James, Witten, Hastie & Tibshirani, "An Introduction to Statistical Learning", ch. 2

Last updated