The same model scores 50% or 100% depending on luck
class A class Bringed = the 8 test points (green right, red wrong)
Which split1
This split's test accuracy
7/8 = 88%
Spread over 200 splits (SD)
0.120
Spread of 5-fold CV (SD)
0.031
True accuracy (1,000 fresh points)
86%
A 5-nearest-neighbour classifier on 40 points. Hold out 8 for testing, train on the other 32, and the test score is 7 of 8, 88%. Drag through other random splits: the same model on the same data scores anywhere from 50% to 100% (red histogram), because 8 points is a tiny test set and each one moves the score by 12.5 points. Its real accuracy, measured on 1,000 fresh points, is 86%. Five-fold cross-validation (green) tests on all 40 points, each exactly once, and its estimate barely moves when you reshuffle: SD 0.031 against 0.120.
Drag through the 200 random splits and watch the red marker jump around the histogram. With 8 test points, every point changes the score by 12.5 percentage points, and which 8 you get is luck. Cross-validation uses every point for testing once, so its estimate depends much less on how the data was shuffled. Both still estimate the accuracy of a model trained on less than all 40 points, which is why the final model is retrained on everything after evaluation.
Step 2 of 6: K-Fold Cross-Validation
Every point is validated exactly once
folds k:
Fold sizes
8
Training points per model
32
Models to train
5
K-fold cross-validation shuffles the data once and deals it into 5 folds (the colours). 40 divides evenly, so every fold has 8 points. Then it trains 5 models. Each one leaves out one fold, trains on the other 32 or so points, and is scored on the fold it never saw. Every point is used for validation exactly once and for training 4 times. More folds means bigger training sets, closer to the full 40, but 5 trainings instead of one.
Switch k and watch the fold sizes. A proper k-fold split shuffles once and deals the points out so that fold sizes differ by at most one; scikit-learn's KFold does exactly this. Larger k trains each model on more of the data, so its estimate is closer to what the final model will do, but it costs k trainings. The k fold models are only used for scoring and are discarded afterwards.
Step 3 of 6: Walking Through the Folds
Train, score, rotate, average
Fold 1: validation accuracy
6/8 = 75%
Mean so far
75%
Spread across folds (SD)
–
True accuracy (dashed)
86%
Fold 1 of 5: a fresh 5-nearest-neighbour model is trained on the 32 faded points, its decision regions are the shading, and it is scored on the 8 ringed points it never saw: 6 right. Step on to the next fold. Note that the five models are thrown away; the one you ship is retrained on all 40 points.
Step through the five folds and compare each fold's score with the dashed true accuracy. Each fold is scored by a model that never saw it. The fold-to-fold spread is large because each fold is small, so it is a poor error bar for the average: the folds share most of their training data, so their scores aren't independent, and no simple formula turns their spread into a trustworthy confidence interval for the mean (Bengio and Grandvalet, 2004).
Step 4 of 6: Stratified K-Fold
Rare classes need to reach every fold
k:
class A class B
Shuffle2
Folds with no class A at all
1 / 5
Class A per fold
0–2
Shuffles (of 50) with an empty fold
49 / 50
A harder dataset: 30 points, only 5 of them class A, as in fraud or disease detection. Random folds shuffle everything together, so class A lands wherever it falls: right now 1 of 5 folds have none, and 49 of 50 shuffles leave at least one fold empty. On those folds recall (the share of class A found) can't even be computed, and accuracy just measures how well the model says "B". With rare classes, use stratified folds; scikit-learn's cross_val_score does this by default for classifiers.
Switch between random and stratified folds and drag the shuffle. With 5 rare examples among 30, random folds almost always leave one fold with none, so recall on that fold is undefined and the fold scores swing. Stratified k-fold splits each class separately, so every fold keeps roughly the overall class ratio. For regression the same idea is applied to binned targets, and for grouped data (several rows per patient) GroupKFold keeps each group in one fold to avoid leakage.
Step 5 of 6: Choosing k
Measured bias, error and cost, up to leave-one-out
k:
Bias: mean (estimate − truth)
+0.1 pts
Typical error (SD)
5.9 pts
Models trained
5
To compare values of k fairly, draw 60 different 40-point datasets from the same source, run k-fold CV on each, and compare its estimate with the true accuracy of the model trained on all 40 (each dot is one dataset; the dashed line is a perfect estimate). With k = 5: bias +0.1 points, typical error 5.9 points, 5 trainings. This is the common default. The striking part: the errors barely depend on k. Most of the error comes from having only 40 points, which no choice of k fixes. So k = 5 or 10 is usually enough, and leave-one-out is mainly worth it when trainings are cheap or data is very scarce.
Switch k and compare the scatter, the bias and the error spread. Each dot is a different dataset drawn from the same source, so this measures what you actually care about: how far the CV estimate lands from the truth. Smaller k is slightly pessimistic because each model trains on less data. Beyond that, the spread barely changes with k here, and the well-known claim that leave-one-out has high variance doesn't show up for this model; whether it does depends on the model and the data (Kohavi, 1995). Repeated k-fold with different shuffles is a cheap way to reduce the part of the noise that comes from the split.
Step 6 of 6: The Leakage Trap
Every learned step goes inside the folds
Noise features200
CV accuracy, selected on all data
84%
CV accuracy, selected inside folds
49%
How much the leak inflates it
+34 pts
The classic way to fool yourself. Here the labels are coin flips and all 200 features are pure noise, so no model can beat 50%. A common pipeline first keeps the 10 features most correlated with the label, then cross-validates a simple classifier on them. Doing that selection on all 40 points lets the validation points vote on which features to keep, and among 200 noise features some line up with the labels by luck. CV then reports 84% on data with no signal at all. More noise features means more lucky ones, so the leak gets worse as you drag right. The rule: every step that learns from data, including scaling, imputing, feature selection and tuning, belongs inside the folds. scikit-learn's Pipeline exists to enforce it.
Switch between selecting features on all the data and inside each fold, then drag the number of noise features. Leakage happens whenever information from the validation points reaches training, here through the feature selection. It's the most common way cross-validation lies, and it also happens with scaling, imputation, oversampling and hyperparameter tuning done before the split. The fix is to wrap every learned step into one pipeline that is refit inside each fold, and to tune hyperparameters with nested cross-validation.
What is cross-validation?
Cross-validation is a way to estimate how well a model will perform on new data by training and testing it several times on different splits of the same dataset. In k-fold cross-validation the data is cut into k equal parts, each part is the validation set once while the other k − 1 parts train the model, and the k scores are averaged. The result is a steadier estimate than any single train/test split can give.
Cross-validation is the standard way to compare models and tune hyperparameters when data is limited, from choosing the regularization strength of a logistic regression to picking the depth of a gradient-boosted tree. Tools such as scikit-learn's GridSearchCV run it automatically for every candidate setting. Medical and scientific studies with a few hundred samples rely on it heavily, because they cannot afford to set aside a large fixed validation set.
Key ideas
A single train/test split gives a noisy estimate that depends on which points land in the test set.
In k-fold cross-validation, every point is validated exactly once and trains the model k − 1 times.
The final score is the mean across folds, reported together with its spread.
Stratified folds keep class proportions stable, and group folds keep related rows together.
Leave-one-out is nearly unbiased but costly, and its variance is not always lower than 10-fold's.
k = 5 or k = 10 is the usual compromise between bias, variance, and compute.
Common questions
Does cross-validation prevent overfitting?
No. Cross-validation measures overfitting rather than preventing it: a model that memorizes its training folds will score poorly on the held-out fold, and the average exposes it. Choosing among many models by their cross-validation score can itself overfit to the folds, which is why a final test set or nested cross-validation is still needed.
Which model do I deploy after k-fold cross-validation?
Usually none of the k fold models. Cross-validation is used to choose the settings, and then one final model is trained on all the available training data with those settings. The cross-validation score is the estimate of how that final model will perform.
How does cross-validation work for time series?
Random folds would let the model train on the future and predict the past. Time series cross-validation uses expanding or sliding windows instead: train on data up to time t, validate on the next block, then move t forward and repeat. scikit-learn implements this as TimeSeriesSplit.
Where should preprocessing happen in cross-validation?
Inside each fold. Scaling, imputation, and feature selection must be fit on that fold's training part only and then applied to its validation part. Fitting them on the full dataset first leaks information from the validation rows and inflates the score.