Bagging vs boosting: many trees side by side or one after another

Bagging and boosting both combine many decision trees, models that predict by asking yes-or-no questions about the input. Bagging trains its trees side by side and averages them. Boosting trains them one after another, and each new tree fixes errors the earlier ones left.

1. 1
   Both rows learn the same 25 training points from the gradient-boosting explainer: a wavy curve plus noise. Every tree is a stump, a tree with a single question: is x below a threshold? It predicts one value on each side. Test error is the mean squared error on 20 points held out from training, the average squared gap between prediction and target. One stump alone scores 0.261.
2. 2
   In bagging, each tree gets a bootstrap sample: 25 random draws with replacement, so some points repeat and others are left out (the hollow dots). Each tree fits its own sample, and the prediction is the average of all 20. No tree depends on another, so they can train in parallel. Averaging smooths out the quirks each tree picks up from its sample, and test error falls to 0.145. It cannot fix a mistake every tree shares. All 20 stumps split between x = −0.25 and 0.63, so their average stays a blunt step.
3. 3
   Boosting starts from the average target and fits tree 1 to the residuals, each point's target minus the current prediction. It adds that tree scaled by 0.3, the learning rate, recomputes the residuals and fits tree 2 to what is left. Each tree needs the one before it, so they train in sequence. Each can split somewhere new, so the sum bends with the curve. After 20 trees the test error is 0.066.