Lensa ML
Lensa ML

Overfitting & Underfitting

Step 1 of 6: Underfitting

When the model is too simple to capture the pattern

xydegree = 1 (linear)
TrainTest
Train error
10.91
Test error
16.47
Both errors are high.

A linear model cannot capture the quadratic pattern in the data. Both training and test errors stay high: this is underfitting. The model has high bias and low variance.

Look at how the straight line misses the curve at both ends. The data comes from y = 0.5x² − 2x + 5 plus noise, with 12 training points and 6 test points. Error here is mean squared error, the average of (y − ŷ)² over the points. For the linear fit, train MSE is about 10.9 and test MSE about 16.5. When even the training error is this high, more data will not help. The model needs more capacity: extra features, a higher degree, or a bigger network.

Step 2 of 6: Overfitting

When the model captures noise instead of signal

xydegree = 11 (high)
TrainTest
Train error
0.00
Test error
20.37

A degree-11 polynomial passes through all 12 training points (near-zero train error) but bends between them. On the test points, its error is well above a simple quadratic's. This is overfitting: the model fit the noise, not the pattern.

Compare the curve's path between training points with where the test diamonds sit. A degree-11 polynomial has 12 coefficients, the same as the number of training points, so it can pass through every one exactly. That is interpolation, and train error drops to zero. The same risk exists in deep learning: Zhang et al. (2016) showed that standard image networks can reach zero training error even on randomly shuffled labels. A model with enough parameters can memorize, so a low training error alone proves nothing.

Step 3 of 6: The Sweet Spot

Just enough complexity to generalize

xydegree = 2sweet spot
TrainTest
Train error
0.37
Test error
0.32
Model complexitydegree = 2
Simple (1)Complex (11)

Drag the slider to find the sweet spot. At degree 2-3, the fit matches the true quadratic without chasing noise. Test error is lowest at degree 2, where the model generalizes best.

Drag the model complexity slider from degree 1 to 11 and watch both error badges. The best degree here is 2, which matches the true function, with a test MSE near 0.32. In real projects the true function is unknown, so the degree, or any similar setting, is chosen by validation error. Keep the test set out of that search. If you tune against the test set, its error stops being an honest estimate. With only 6 test points, as here, the estimate is also noisy.

Step 4 of 6: Bias vs Variance

Two sources of error, one tradeoff

Bullseye = true valueHigh Bias, Low Variance

Bias means consistently wrong (off-center). Variance means inconsistent (scattered). The goal is low bias and low variance: predictions that are both accurate and consistent.

Click each button and compare where the shots land. For squared error, the expected test error splits into three parts: bias², variance, and irreducible noise. Bias is how far the average prediction, over many possible training sets, sits from the truth. Variance is how much predictions change from one training set to the next. The noise term is a floor no model can beat. Simple models usually have high bias, and flexible models have high variance. Averaging many high-variance models, as random forests do, cuts variance.

Step 5 of 6: Model Complexity Curve

The U-shaped test error

underfitsweet spotoverfitModel Complexity (polynomial degree)Error (log scale)Overfit
Train errorTest error
Complexitydegree = 5
Train error
0.11
Test error
0.59

The classic U-shaped test error curve. Train error keeps falling as the degree rises, while test error bottoms out at degree 2 and then climbs unevenly. The gap between curves is the generalization gap.

Drag the complexity slider and follow the two curves on the log scale. The textbook picture is a smooth U, but small datasets give bumpy curves, as here: test error peaks at degree 10 and partly recovers at degree 11. Belkin et al. (2019) described a related effect called double descent. Past the point where a model can fit the training data exactly, test error can fall again as capacity grows. Very large neural networks often operate in that regime.

Step 6 of 6: What To Do About It

Practical strategies for finding the balance

Is test error high?YesIs training error also high?YesNoUnderfittingOverfitting
Solutions:
  • Use a more complex model
  • Add more features
  • Reduce regularization
  • Train longer / more epochs
Solutions:
  • Get more training data
  • Add regularization (L1/L2)
  • Use dropout / early stopping
  • Simplify the model
Key Principle
Use a validation set to detect the problem early, then apply the right strategy from above.

Diagnose first, then act. If both errors are high, the model is too simple. If only test error is high, the model is too complex. A validation set catches both problems during training.

Follow the decision tree from the top question down. The two diagnostics are training error and validation error. A useful extra tool is the learning curve, which plots both errors against training set size. If the curves have converged at a high error, the model underfits and more data will not help. If a wide gap remains between them, more data usually will. Early stopping, dropout, and L2 regularization are cheap to try. Collecting more data is often the most effective fix.

What is overfitting in machine learning?

Overfitting is when a model fits its training data so closely, noise included, that it performs worse on new data. Its opposite, underfitting, is when a model is too simple to capture the real pattern and does poorly on both training and new data. The goal of training is a model that generalizes: low error on examples it has never seen.

Every supervised model faces this tradeoff, from linear regression to large neural networks. Practitioners detect it by holding out a validation set and comparing training and validation error. The fixes include more data, regularization such as weight decay and dropout, early stopping, data augmentation, and choosing a model of the right size.

Key ideas

  • Underfitting means high error on both training and test data.
  • Overfitting means low training error but much higher test error.
  • A model with as many parameters as data points can fit the training set exactly.
  • Expected test error splits into bias², variance, and irreducible noise.
  • Model settings should be chosen with validation data, never with the test set.
  • More data, regularization, and early stopping are the standard remedies for overfitting.

Short takes on this topic

Common questions

How do I know if my model is overfitting?

Compare training and validation performance. If training error keeps falling while validation error flattens or rises, the model is overfitting. A large and growing gap between the two curves is the clearest sign.

Is some overfitting acceptable?

Yes. Training error is almost always lower than validation error, and a small gap is normal. What matters is validation performance. A model with a larger gap but lower validation error is still the better model.

Can deep neural networks overfit?

They can, and with enough parameters they can memorize any training set. In practice, large networks trained with weight decay, data augmentation, and early stopping often generalize well despite having far more parameters than examples, which classical theory did not predict.

Does more data always reduce overfitting?

More data from the same distribution reduces variance, so it usually shrinks the gap between training and validation error. It does not fix underfitting, and it does not help if the new data is noisy, mislabeled, or drawn from a different distribution than the test data.

Related explainers

References

  1. James, Witten, Hastie & Tibshirani, "An Introduction to Statistical Learning", chapter 2
  2. Goodfellow, Bengio & Courville (2016), "Deep Learning", section 5.2: Capacity, Overfitting and Underfitting
  3. Zhang et al. (2016), "Understanding deep learning requires rethinking generalization"
  4. Belkin et al. (2019), "Reconciling modern machine learning practice and the bias-variance trade-off"

Last updated