1Model Evaluation & Metrics 2Train, Validation & Test Sets 3Cross-Validation 4Bias-Variance Tradeoff 5Learning Curves 6Class Imbalance & Resampling 7Feature Selection 8Curse of Dimensionality

Lensa ML

Train, Validation & Test Sets

Step 1 of 6: Why Split?

You can't grade your own exam

Model complexity (polynomial degree)1

Degree

Train MSE

0.612

Params

A simple model follows the general trend without bending toward individual points. Higher degrees only shave a little more off its training error.

Drag the polynomial degree from 1 to 10 and watch the TRAIN MSE card. It only falls from about 0.61 to 0.49, because a least-squares fit on the same 30 points never gets worse when a degree is added. More parameters always help on data the model has already seen, so the training error cannot rank models. The true curve here is the quadratic 2 + 1.2x − 0.18x², plus noise. Any bend beyond degree 2 is fitting that noise, and only data kept out of fitting can reveal it.

Step 2 of 6: Train & Test

The basic split

Train/Test ratio80/20

Model degree3

Train size

Test size

Train MSE

0.46

Test MSE

1.32

A 80/20 split is standard. The test set gives an unbiased estimate of how the model will perform on new, unseen data.

Drag the Train/Test ratio between 50/50 and 90/10, then change the model degree. With 30 points, a 90/10 split leaves 3 test points, and one unlucky point can swing the test MSE a lot. Both sets must come from the same distribution, so the split is random or spread across the range of x, as it is here. Time series are the exception: train on the past and test on the future, because a random split lets the model see later values. With a million examples, even a 99/1 split leaves 10,000 test cases.

Step 3 of 6: The Validation Set

Tuning without cheating

Validation fraction15%

Model degree3

Train MSE

0.40

Val MSE

2.15

Test MSE

0.81

The validation set is your practice exam. You use it to tune hyperparameters (like model complexity) without touching the test set. A 15% validation split is typical. The test set stays locked away until final evaluation.

Raise the validation fraction from 10% to 30% while the test share stays at 20%, and watch the training share shrink from 70% to 50%. Training data sets the weights. Validation data picks the settings that training cannot learn: the degree here, or learning rate, layer count, and regularization strength in a neural network. Each choice made by looking at validation scores fits the model slightly to the validation set, so the best validation score ends up a little optimistic. That bias is the reason the test set stays separate.

Step 4 of 6: Underfitting & Overfitting

What the train and validation curves look like

Model complexity3

Train loss

0.608

Val loss

0.903

Gap

0.295

Underfitting: Both train and val loss are high. The model is too simple to capture the pattern. Increase complexity.

Drag the model complexity from 1 to 10 and follow the two curves. Validation loss is lowest at complexity 5 (0.608) and rises after that, while training loss keeps falling to 0.067 at 10. The diagnosis rule is simple: high training loss means underfitting, and low training loss with a growing gap means overfitting. Neural networks show the same shape over training time instead of model size. Early stopping uses it by saving the weights from the epoch with the lowest validation loss rather than the last epoch.

Step 5 of 6: Cross-Validation

Using all data for validation

Number of folds (k)k = 5

Folds

Avg score

74.1%

Std dev

3.7%

K-fold cross-validation rotates through 5 folds so every data point is validated exactly once. (The fold scores here are illustrative, not from a fitted model.) 5-fold is the most common choice. The std of 3.7% across folds shows how much the score depends on the split.

Drag the number of folds k from 2 to 10. Each row trains a fresh model on k − 1 folds and scores it on the one yellow fold, so k = 10 trains ten models. Cross-validation replaces the single validation split when data is scarce, but it does not replace the test set. After choosing settings this way, the usual step is to retrain one model on all non-test data and score it once on the test set. Nested cross-validation adds an outer loop when data is too scarce for a fixed test set.

Step 6 of 6: The Golden Rule

Never touch the test set until the end

Times test set was "peeked" at0

Reported

75.0%

Real-world

75.0%

Inflation

The golden rule: never touch the test set until the final evaluation. Train on training data, tune on validation data, and evaluate on test data once. (The scores in this step are illustrative.)

Drag the peek counter from 0 to 20 and compare the Reported and Real-world bars. A peek is any decision made after seeing a test score: changing a feature, a threshold, or which model to keep. Each one fits the choice to those particular test examples. Leakage also has quieter forms. Scaling features with statistics from the full dataset before splitting, or letting near-duplicate images land in both sets, inflates test scores the same way. Benchmarks suffer from it when many papers tune against one test set for years.

What are training, validation, and test sets?

Training, validation, and test sets are three disjoint parts of a dataset, each with one job. The training set fits the model's parameters, the validation set chooses settings such as model size or learning rate, and the test set gives a single final estimate of performance on data the model has never influenced. Keeping them separate is what makes a reported score mean anything.

Every supervised learning project uses this split, from a scikit-learn regression to a large neural network. Kaggle competitions hide a private test set for the same reason, and public benchmarks such as ImageNet keep their test labels apart. When the split is done badly, through leakage, duplicates, or repeated tuning on the test set, a model can look excellent in the lab and fail in production.

Key ideas

The training score always improves with complexity, so it cannot be used to choose a model.
The training set fits parameters, the validation set chooses hyperparameters, and the test set reports the final score.
All three sets must come from the same distribution, except in time series, where the test set comes from the future.
Validation loss shaped like a U marks the move from underfitting to overfitting.
Cross-validation reuses data for validation when samples are scarce.
Any decision based on test results turns the test set into a second validation set.

Common questions

What is a good train/validation/test split ratio?

70/15/15 and 80/10/10 are common defaults for datasets of a few thousand to a few hundred thousand examples. What matters is the absolute size of the test set: with 1,000 test examples and 90% accuracy, the standard error is about √(0.9 × 0.1 / 1000) ≈ 0.95 percentage points. Very large datasets can give 98% or more to training and still have precise validation and test scores.

What is the difference between a validation set and a test set?

Both are held out from training, but the validation set is used many times to compare models and settings, while the test set is used once at the end. Because the validation set influences decisions, its score is biased upward. The test set's score is unbiased only if nothing was chosen based on it.

What is a stratified split?

A stratified split keeps each class's share the same in every set. With 5% positives, a random split of a small dataset can leave the test set with almost no positives, while stratification guarantees about 5% in each part. Libraries such as scikit-learn support it through a stratify argument.

Can I train on the validation set after tuning?

Yes. Once the hyperparameters are fixed, retraining on the training and validation data together usually gives a slightly better model, because it sees more examples. The test set is still kept out, and it scores this final retrained model.

Related explainers

Overfitting & Underfitting — The bias-variance tradeoff and generalization
Cross-Validation — K-fold, stratified, and leave-one-out evaluation
Model Evaluation & Metrics — Precision, recall, F1, ROC curves, and regression metrics
Hyperparameters — The knobs you tune before training begins
Learning Curves — Diagnosing underfitting and overfitting from training size
Bias-Variance Tradeoff — Why models underfit or overfit — the fundamental decomposition

References

Last updated October 25, 2026