ENID
Lensa ML

Gradient Boosting

Step 1 of 6: Start with a Constant

The initial prediction

Initial prediction0.08
-3-2-10123-2-1012xy
Prediction
0.08
MSE
1.483
Residual range
-1.6…1.6
Points
25

Gradient boosting starts with a simple prediction, usually the mean of the targets. You're close to the mean, the constant that minimizes squared error.

Drag the initial prediction slider and watch the MSE card. The constant that minimizes squared error is the mean of the targets, about 0.08 for these 25 points. Other losses start from other constants: the median for absolute error, and the log-odds of the positive class for log loss in binary classification. This starting value is the model's f_0(x). Every later tree adds a correction on top of it, so the final model has the form f(x) = f_0 + η·Σ h_m(x), where h_m is the m-th tree.

Step 2 of 6: Fit the Residuals

A tree learns the errors

Round 1
Prediction vs Dataf(x)Residuals (what the next tree learns)0-3-2-10123
Round
1
MSE before
1.483
MSE after
0.822
Improvement
44.5%

The top curve is the ensemble's current prediction. The red dots below are the residuals, the errors it hasn't captured yet, and the colored step is the new tree fitted to them.

Press Play or drag the round slider, and compare the residuals in the bottom panel with the stump drawn over them. With squared-error loss L = ½(y − f)², the negative gradient with respect to the prediction f is y − f, the residual. Fitting a tree to the residuals is therefore a gradient descent step in function space, the view Jerome Friedman set out in 2001. For other losses, such as log loss, the tree fits pseudo-residuals computed from that loss's gradient. Each stump in this demo tries every split point and keeps the one with the lowest squared error.

Step 3 of 6: Learning Rate

Small steps, big results

Learning rate η0.30
Round1
1. Tree fitted to residuals-1.011.28× η = 0.302. Scaled update (η × tree)-0.3030.385add to ensemble ↓3. Updated prediction-3-1013
η
0.30
Round
1
MSE before
1.483
MSE after
0.822

η=0.30 scales each tree's contribution to 30% of its raw values, trading learning speed for stability.

Drag the learning rate η and round sliders, then follow the three panels from the raw tree to the scaled update. Multiplying each tree by η, called shrinkage, means every tree fixes only part of the error, which leaves later trees something to learn. Friedman found that small values, around η ≤ 0.1, consistently gave lower test error at the cost of more trees. Libraries usually default to 0.1 or 0.3. Shrinkage is often paired with row subsampling, where each tree sees a random fraction of the data, a variant called stochastic gradient boosting.

Step 4 of 6: Building Sequentially

The ensemble grows

R1
-3-2-10123-2-1012xy
Trees
1
Train MSE
0.822
Test MSE
0.718
Resid. std
0.907

Each tree corrects the ensemble's mistakes, so predictions improve step by step. Early rounds capture the big patterns in the data.

Press Play or drag the round slider from 1 to 30 and compare the train and test MSE cards. Boosting and bagging reduce different errors. Bagging averages independent high-variance trees to cut variance, while boosting adds small high-bias trees one after another to cut bias. That is why boosting uses shallow trees, often depth 3 to 8, or one-split stumps as here. The trees cannot be trained in parallel, because each one depends on the trees before it, so libraries parallelize the split search inside each tree instead.

Step 5 of 6: Trees × Learning Rate

The tradeoff

-3-2-10123-2-1012xy
η
0.30
Trees
20
Train MSE
0.012
Test MSE
0.066

A low learning rate with many trees can match a high rate with few trees, and it usually generalizes better. Adjust both sliders to see how they interact.

Drag the learning rate η and max trees sliders together. On this data, η = 1 with 5 trees reaches a test MSE of about 0.19, while η = 0.05 with 50 trees reaches about 0.07. Halving η roughly doubles the number of trees needed for the same training fit. The usual recipe is to fix a small learning rate, such as 0.05 or 0.1, and let early stopping pick the number of trees. Tree depth, the minimum samples per leaf, and L1 or L2 penalties on leaf values, as in XGBoost, are the other main controls.

Step 6 of 6: Learning Curve

Watching loss descend

R1
01020304050Rounds0.00.20.40.60.81.01.21.41.6MSEbestTrainTest
Round
1
Train loss
0.916
Test loss
0.801
Gap
-0.116

Train loss keeps falling. Test loss bottoms out at 0.050 (round 19) and creeps back up to 0.056 by round 50. That rise is overfitting; here it is too small to see on the chart, so read it from the Test loss card. Both losses are falling. The best test loss comes at round 19.

Press Play or drag the round slider to 50 and find the gold best marker. Here the test loss bottoms out at round 19, with an MSE of about 0.050, and creeps back up to 0.056 by round 50 while the training loss keeps falling to about 0.006. Early stopping turns this into a rule: hold out a validation set, track its loss after every round, and stop once it has not improved for a set number of rounds, such as 50. XGBoost and LightGBM support this directly. The final model keeps only the trees up to the best round.

What is gradient boosting?

Gradient boosting is an ensemble method that builds a strong model by adding many weak ones, usually small decision trees, one at a time. Each new tree is fitted to the errors of the current ensemble, more precisely to the negative gradient of the loss, and its output is scaled down by a learning rate before being added. With squared-error loss, that gradient is simply the residual: the target minus the current prediction.

Gradient-boosted trees are among the most accurate models for tabular data and often win tabular machine learning competitions. XGBoost, LightGBM, and CatBoost are the common implementations. They power click-through and ranking models in search and advertising, as well as credit scoring, fraud detection, and demand forecasting.

Key ideas

  • Gradient boosting adds trees one at a time, each fitted to the current ensemble's errors.
  • Each tree fits the negative gradient of the loss, which for squared error is the residual.
  • A learning rate scales every tree's contribution so that no single tree dominates.
  • A smaller learning rate needs more trees but usually generalizes better.
  • Boosting reduces bias with shallow trees, while bagging reduces variance with deep ones.
  • Early stopping on a validation set chooses the number of trees.

Short takes on this topic

Common questions

What is the difference between gradient boosting and AdaBoost?

AdaBoost reweights the training points after each round so that the next model focuses on the examples that were misclassified. Gradient boosting instead fits each new model to the gradient of a loss function. AdaBoost turns out to be a special case of this framework with exponential loss, and gradient boosting extends the idea to any differentiable loss.

What makes XGBoost different from plain gradient boosting?

XGBoost uses second-order (Newton) steps, adds L1 and L2 penalties on leaf values, learns a default direction for missing values at each split, and uses a fast approximate split search. Chen and Guestrin described it in 2016. LightGBM speeds training further with histogram-based splits and leaf-wise tree growth.

Can gradient boosting be used for classification?

Yes. For binary classification it boosts the log-odds under log loss, and each tree fits the gradient y − p, where p is the current predicted probability. For K classes it usually grows K trees per round, one per class score, and combines the scores with softmax.

Does gradient boosting need feature scaling?

No. Its trees split on thresholds, so a monotonic rescaling of a feature leaves the splits unchanged. It is also less sensitive to outliers in the inputs than linear models are, although outliers in the target still matter under squared-error loss.

Related explainers

References

  1. Friedman (2001), "Greedy Function Approximation: A Gradient Boosting Machine", Annals of Statistics
  2. Friedman (2002), "Stochastic Gradient Boosting", Computational Statistics & Data Analysis
  3. Chen & Guestrin (2016), "XGBoost: A Scalable Tree Boosting System"
  4. James, Witten, Hastie & Tibshirani, "An Introduction to Statistical Learning", chapter 8: Tree-Based Methods

Last updated