ENID
Lensa ML

Learning Curves

Step 1 of 6: What is a Learning Curve?

Accuracy vs. training size

0%20%40%60%80%100%1058105153200Training SizeAccuracyTrainValidationillustrativeN=80
N
80
Train acc
92.8%
Val acc
79.5%
Gap
13.3%

A learning curve plots model performance against training-set size. The gap between the training and validation curves shows whether the model underfits or overfits.

Drag the training size N from 10 to 200 and watch both points move along their curves. To build a learning curve on a real project, train the same model on growing subsets of the training data, say 10%, 20%, and so on up to 100%, and score each on one fixed validation set. On real data, training error usually starts near zero, since a few points are easy to fit, and rises with N, while validation error falls. The demo follows that pattern: at N = 10 training accuracy is 98.8% and validation accuracy 56.7%, and by N = 200 they are 88.7% and 86.2%. Where the two curves settle tells you which problem you have.

Step 2 of 6: High Bias Pattern

Both curves plateau low

Data follows a curve: can a degree-1 polynomial capture it?
x (feature)y (target)true patterndegree-1 fit
0%20%40%60%80%100%1058105153200Training SizeAccuracyTrainValidationillustrativeplateau
Degree
1
Fit error
0.025
Train acc
55.0%
Plateau
55.0%

A straight line can't follow this curved pattern, so the fit error stays high. No amount of extra data fixes a model class that is too rigid.

Set the model degree to 1, 2, and 3 and compare the Fit error card. The line leaves an error of about 0.025, while the quadratic and cubic both reach about 0.002, close to the noise added to these 20 points. The true pattern is 0.3 + 0.5·sin(πx) on [0, 1], a single hump that a parabola approximates well. The learning-curve signature of high bias is two curves that meet early at a poor score and stay flat, as both do at 55% for the line. Doubling the data moves neither of them.

Step 3 of 6: High Variance Pattern

A persistent gap

Degree-4 polynomial through 5 training points: memorizes train, misses validation
x (feature)y (target)Train (5)Val (10)overfit model
0%20%40%60%80%100%1058105153200Training SizeAccuracyTrainValidationillustrative28% gap
Train err
0.0000
Val err
141.524
Err ratio
1.4M×
Gap
28.2%

Training error stays at zero because the degree rises with the sample count, while the curve swings far from the validation points to the right of the last training point.

Drag the training samples slider from 3 to 10. The demo fits a polynomial of degree n − 1 through n training points, so training error is always zero: with as many coefficients as points, an exact fit exists. Validation error at 5 to 8 samples runs into the hundreds, because the curve swings far outside the [0, 1] range to the right of the last training point. At 10 evenly spread samples it drops to about 0.02. Fixed-degree models behave the same way as N grows past the number of parameters.

Step 4 of 6: Adding More Data

Does it help?

0%20%40%60%80%100%10133255378500Training SizeAccuracyTrainValidationillustrativeN=80
N
80
Gap
2.8%
Trend
closing
Diagnosis
High Bias

The curves have met at a low accuracy: the model's limit, not the data, is holding it back. Once the validation curve is flat, more data won't help; a more flexible model will.

Drag N up to 500 and switch the model complexity between simple and complex settings. Learning curves answer a budget question before anyone collects data. If validation accuracy is still climbing at the largest N, more data will probably pay off; if it has flattened, it will not. Large studies of deep networks find that test error falls as a power law in dataset size over a wide range, which is a straight line on a log-log plot. Teams extrapolate that line to estimate how much data a target accuracy needs.

Step 5 of 6: Changing Model Complexity

The other lever

0%20%40%60%80%100%1058105153200Training SizeAccuracyTrainValidationillustrativeN=100Good Fit
Complexity
5
Train acc
91.7%
Val acc
81.9%
Gap
9.8%

Good range: validation accuracy is near its peak here. Past this point, extra complexity starts to cost validation accuracy.

Drag the model complexity from 1 to 10 with N fixed at 100. This step reads the curves the other way: one learning curve per complexity, compared at a single N. Plotting validation score against complexity instead of N gives a validation curve, which is scikit-learn's name for the companion plot. The two plots separate the levers. A validation curve picks the capacity, and a learning curve then shows whether that capacity is starved for data.

Step 6 of 6: Diagnose Your Model

Interactive scenario explorer

0%20%40%60%80%100%1058105153200Training SizeAccuracyTrainValidationillustrativeGood
Diagnosis
Good
Action
Model is balanced
Train acc
92.8%
Val acc
79.5%

Balanced: the gap is small and validation accuracy is not held back by underfitting. Bias and variance are both under control.

Move both sliders and watch the Diagnosis and Action cards. The rule here uses two thresholds: validation accuracy below 70% with a gap under 12 points reads as underfitting, and a gap above 15 points reads as overfitting. Real projects add a third reference, the best achievable score, such as human-level accuracy. The distance from that score to training performance is avoidable bias, and the distance from training to validation performance is variance. Whichever is larger is the one to work on first.

What is a learning curve in machine learning?

A learning curve is a plot of a model's training and validation performance against the amount of training data. Reading where the two curves sit and how far apart they are tells you whether a model suffers mainly from high bias (underfitting) or high variance (overfitting), and whether collecting more data is likely to help.

Learning curves are a standard diagnostic before spending money on data labeling or bigger models. A team deciding whether to label another 50,000 images, a Kaggle competitor choosing between a deeper network and more augmentation, and a researcher fitting scaling laws for language models all use the same plot. The term also appears in a looser sense for loss plotted against training epochs, which answers a different question.

Key ideas

  • A learning curve plots training and validation performance against training-set size.
  • High bias shows as two curves that converge early at a poor score.
  • High variance shows as a large gap between training and validation performance.
  • More data helps high variance but does little for high bias.
  • A flat validation curve at the largest N means more data is unlikely to pay off.
  • Validation curves vary complexity at fixed N and complement learning curves.

Common questions

What is the difference between a learning curve and a loss curve?

A learning curve in the strict sense varies the amount of training data. A loss curve, often also called a learning curve in deep learning, plots loss against training epochs or steps for one fixed dataset. The loss curve is used for early stopping and learning-rate tuning, while the data-size curve is used to decide whether to collect more data.

How do I plot a learning curve in scikit-learn?

Use sklearn.model_selection.learning_curve, which takes an estimator, the data, and a list of training sizes. It runs cross-validation at each size and returns training and validation scores for every fold. Plot the mean of each with a shaded band of one standard deviation.

Why is my validation accuracy higher than my training accuracy?

Common causes are regularization that is only active during training, such as dropout, and data augmentation that makes training examples harder than validation ones. Training accuracy logged as a running average over an epoch also lags behind the model's end-of-epoch weights. A small or unusually easy validation set can produce the same effect.

How many training sizes should a learning curve use?

Five to ten sizes are usually enough, spaced on a log scale such as 1%, 3%, 10%, 30%, and 100% of the data. Log spacing shows the early rise and the later plateau equally well. Repeat each size with cross-validation or several seeds, because small subsets give noisy scores.

Related explainers

References

  1. Perlich, Provost & Simonoff (2003), "Tree Induction vs. Logistic Regression: A Learning-Curve Analysis", JMLR
  2. Hestness et al. (2017), "Deep Learning Scaling is Predictable, Empirically"
  3. Kaplan et al. (2020), "Scaling Laws for Neural Language Models"

Last updated