1Decision Trees & Splitting 2K-Nearest Neighbors 3Support Vector Machines 4Naive Bayes Classifier 5K-Means Clustering 6Bagging & Random Forests 7Gradient Boosting 8Ensemble Methods & Stacking

Lensa ML

Ensemble Methods & Stacking

Step 1 of 6: Ensembles Need Different Mistakes

Three different models vs three near-copies

depth-2 tree · 82.4%

logistic regression · 82.0%

1-NN · 76.4%

160 training points from two heavily overlapping moons; accuracies use 1,000 fresh test points. Every model is trained in your browser.

Majority vote

81.5%

Best single model

82.4%

How alike their mistakes are (error correlation)

0.53

Ceiling: test points at least one model gets right

91.0%

A shallow tree cuts the plane into boxes, logistic regression draws one straight line, and 1-nearest-neighbour copies the label of the closest training point. They make different kinds of mistakes: their errors correlate at 0.53. The big panel is their majority vote, 81.5% against an average member's 80.3%. The ceiling card shows what diversity buys: 91.0% of test points have at least one model that gets them right, so a perfect combiner could reach that. For these different models the ceiling is high, but a plain vote doesn't beat the best member (82.4%), because the weak 1-NN has the same say as the others. Learning whom to trust is what stacking adds.

Switch between the two families and compare the error correlation and the ceiling. An ensemble can only fix a point if at least one member gets it right, so the share of points every member gets wrong is a hard limit. Different model families (trees, linear models, nearest neighbours, neural networks) tend to fail in different places; copies of one model trained on slightly different data mostly fail together. Bagging works around that by making trees deliberately unstable, as the random forests explainer shows.

Step 2 of 6: Hard Voting

Where the majority rescues, and where it can't

Points the vote rescues

132

Points the vote loses (one model was right)

Majority vote accuracy

81.5%

With three voters, the majority is right exactly when at most one of them is wrong. For the three different models, 132 test points have one model wrong and the vote fixes them; 95 have two wrong, so the one model that was right gets outvoted; and 90 are wrong for everyone. Voting helps when the "one wrong" bar is tall compared with the "two wrong" bar, which happens when each model is decent and their mistakes are spread out. With two classes and three voters there can never be a tie, one reason odd ensemble sizes are common.

Switch families and compare the bars. A majority vote is right when at most half the members are wrong. If each of n independent members were right with probability p > ½, the vote would get more accurate as n grows (Condorcet's jury theorem), but real models aren't independent, so the gain is limited by how often they fail together.

Step 3 of 6: Soft Voting

Averaging probabilities, and why calibration matters

x₁-0.62

x₂-0.47

Hard vote (count the yeses)

amber

Soft vote (average probability)

0.49 → blue

Test points where they disagree (ringed)

Hard · soft accuracy

81.5% · 80.8%

Hard voting counts verdicts; soft voting averages the probabilities. Here they disagree: 2 models lean amber, so the hard vote says amber, but depth-2 tree says 0.82, logistic regression says 0.64, 1-NN says 0.00, and the average, 0.49, says blue. A confident model can outweigh two unsure ones. Soft voting uses more information, but it trusts every model's probabilities equally, and they aren't equally reliable: 1-NN only ever says 0 or 1, so it is always "confident". On this data that drags soft voting slightly below hard voting, 80.8% against 81.5%. Probabilities need to be calibrated before averaging them means much.

Drag the cross to a ringed point and compare the two verdicts. Soft voting averages predicted probabilities, so a confident model can overrule two hesitant ones. It works best when the probabilities are calibrated, meaning a stated 0.8 is right about 80% of the time. Overconfident models such as 1-NN or deep, unpruned trees can be calibrated first, for example with Platt scaling or isotonic regression.

Step 4 of 6: Stacking: Level 0

Out-of-fold predictions from every base model

Folds done

5 / 5

Out-of-fold accuracy: depth-2 tree

79.4%

Out-of-fold accuracy: logistic regression

83.8%

Out-of-fold accuracy: 1-NN

71.9%

Stacking trains a second model to combine the first ones, and it needs honest examples to learn from. So level 0 runs 5-fold cross-validation: for each fold, the three base models are trained on the other four folds and predict the held-out one. After all five folds, every row has three predictions made by models that never saw it. The out-of-fold accuracies show how each model really does on unseen points: depth-2 tree 79.4%, logistic regression 83.8%, 1-NN 71.9%. This 160 × 3 table, plus the labels, is the training set for level 1.

Step through the folds and watch the table fill in. Each prediction comes from base models that didn't see that row, so the table records how they behave on new data. scikit-learn's StackingClassifier does this with cross_val_predict. The out-of-fold models are only used to build the table; the base models used at prediction time are retrained on all the training data.

Step 5 of 6: Stacking: Level 1

A meta-model learns whom to trust

Stacked model, test accuracy

83.1%

Best single model

82.4%

Weight on 1-NN

-0.08

Level 1 is a logistic regression whose inputs are the three base models' out-of-fold probabilities (as log-odds). It learns how much to trust each: depth-2 tree 0.21, logistic regression 0.86, 1-NN -0.08. It leans on logistic regression, uses the tree a little and all but ignores 1-NN, whose out-of-fold record is the worst. The final stack refits the base models on all 160 points and combines them with these weights: 83.1% on test data, above every base model (best 82.4%). Switch to the leaky version to see why the out-of-fold step matters.

Switch between out-of-fold and in-sample training and compare the weights. Wolpert (1992) introduced stacking with exactly this rule: the combiner must learn from predictions made on data the base models didn't train on. A simple, regularised meta-model such as logistic regression is usually best, because the level-1 training set is small and a flexible meta-model would overfit it.

Step 6 of 6: Comparing Strategies

Every combination on the same test set

Best single model

82.4%

Best combination

83.1%

Gain from combining

+0.7 pts

All the strategies on the same 1,000 test points (the dashed line is the best single model). With three different models, plain voting lands between the members, and stacking, which learned to down-weight 1-NN, beats the best one by 0.7 points. The leaky stack collapses to 1-NN. The gains here are small, which is typical: an ensemble is worth it when its members are individually good, genuinely different, and combined with weights learned honestly. That's why winning competition entries stack very different model families (boosted trees, neural networks, linear models) rather than many copies of one.

Switch families and compare all strategies against the dashed best single model. Typical gains from ensembling are a few points at most, and they cost extra training, latency and complexity. They matter most in competitions and high-stakes settings; elsewhere a single well-tuned model is often the better engineering choice.

What is stacking in ensemble learning?

Ensemble learning combines the predictions of several models to get a result that is more accurate and more stable than any one of them. The simplest combiners are voting rules: hard voting takes the majority class, and soft voting averages predicted probabilities. Stacking, or stacked generalization, goes further and trains a second-level model, the meta-learner, to decide how to combine the base models' predictions.

Ensembles work because different models make different mistakes, and combining them cancels some of those errors. Stacked ensembles of gradient-boosted trees, neural networks, and linear models are a common recipe in machine learning competitions, and AutoML systems such as AutoGluon and H2O build stacked ensembles automatically. The cost is extra training time and a model that is harder to explain and to deploy.

Key ideas

An ensemble combines several models so that their individual errors partly cancel.
Ensembles gain the most when the base models are accurate and make different mistakes.
Hard voting counts predicted labels, while soft voting averages predicted probabilities.
Stacking trains a meta-learner on the base models' out-of-fold predictions.
Out-of-fold predictions keep the meta-learner from trusting overfit base models.
A simple, regularized meta-learner such as logistic regression is usually enough.

Common questions

What is the difference between stacking and blending?

Blending is a simpler version of stacking. Instead of out-of-fold predictions from cross-validation, it holds out a single validation set, trains the base models on the rest, and fits the meta-learner on their predictions for the holdout. It is faster and easier to implement, but the meta-learner learns from less data.

How is stacking different from bagging and boosting?

Bagging and boosting usually combine many models of the same type, trained on resampled data or on the errors of earlier models. Stacking combines different model types and learns the combination from data. The approaches mix well: a stacked ensemble often uses a random forest and a boosted model as two of its base learners.

Which models make good base learners for stacking?

Models that are each reasonably accurate but work in different ways, for example gradient-boosted trees, k-nearest neighbors, a neural network, and a regularized linear model. Diversity matters more than adding another strong model of the same kind, because similar models make the same mistakes.

Does stacking always improve accuracy?

No. If the base models are highly correlated, or the dataset is small, a stacked ensemble may do no better than the best single model or a simple average. It also multiplies training and inference cost, so the gain has to justify the added complexity.

Related explainers

Bagging & Random Forests — Bootstrap aggregation and decorrelated tree ensembles
Gradient Boosting — Sequential residual fitting for powerful predictions
Cross-Validation — K-fold, stratified, and leave-one-out evaluation
Decision Trees & Splitting — Recursive partitioning, Gini impurity, and pruning
Model Evaluation & Metrics — Precision, recall, F1, ROC curves, and regression metrics
Overfitting & Underfitting — The bias-variance tradeoff and generalization

References

Last updated November 4, 2026