Gradient Boosting
Start with a Constant
The initial prediction
Gradient boosting starts with a simple prediction — usually the mean of the targets. You're close to the mean — a good starting point that minimizes the initial squared error.
Step 1 of 6: Start with a Constant
The initial prediction
Gradient boosting starts with a simple prediction — usually the mean of the targets. You're close to the mean — a good starting point that minimizes the initial squared error.
Step 2 of 6: Fit the Residuals
A tree learns the errors
The green curve (top) shows the ensemble's current prediction. The pink dots (bottom) are the residuals — errors the ensemble hasn't captured yet. The coloured step function is the new tree fitted to those errors.
Step 3 of 6: Learning Rate
Small steps, big results
η=0.30 scales each tree's contribution to 30% of its raw values. This balances learning speed with stability.
Step 4 of 6: Building Sequentially
The ensemble grows
Each tree corrects the ensemble's mistakes — predictions improve incrementally. Early rounds capture the big patterns in the data.
Step 5 of 6: Trees × Learning Rate
The tradeoff
Low learning rate + many trees ≈ high learning rate + few trees, but the former generalizes better. Adjust both sliders to see how they interact.
Step 6 of 6: Loss Landscape
Watching loss descend
Train loss always decreases; watch the test loss — when it rises, you're overfitting. Both losses are dropping. The best test loss is at round 8.