How do gradient-boosted decision trees work, and how do they differ from random forests?
Answer
Gradient boosting builds an additive ensemble sequentially: each new tree is fit to the negative gradient of the loss with respect to the current ensemble’s predictions (for squared error, the residuals), then added in with a small learning rate. The ensemble is a directed attack on bias; every stage shrinks whatever error remains. A random forest does the opposite: it grows many deep trees independently on bootstrap samples with random feature subsets and averages their votes, which attacks variance. So boosting uses shallow trees, a learning rate, and sequence, while bagging uses deep trees, randomness, and parallel averaging.
(1) Sequential Residual Fitting: tree m targets the current gradient (pseudo-residuals); with squared loss this is literally y minus the running prediction, and shrinkage keeps any single tree from dominating.
(2) Bias vs Variance: boosting reduces bias stage by stage but can chase noise if run long or deep; random forests reduce variance by decorrelating many overfit trees, but their bias is roughly that of one tree.
(3) Modern Refinement: Yandex showed classic gradient boosting leaks target information into its own gradients (prediction shift), and CatBoost’s ordered boosting fixes it by computing each example’s gradient from a model trained only on earlier examples in a random permutation.
Figure 1: Stage-wise fitting: each new tree targets the residuals (orange stems) left by the current ensemble, so the prediction curve (blue) bends toward the data a little at every stage.
Structurally, the two ensembles could not be more different. A random forest is embarrassingly parallel: tree one and tree one hundred do not know each other, so you can train them on separate machines and the average cancels their individual overfitting. Gradient boosting is an inherently sequential chain: tree one hundred is defined by the mistakes of the previous ninety-nine, which buys accuracy on structured tabular data but makes training serial and makes the model sensitive to label noise, because late-stage trees are explicitly trained to fit whatever is left, including the noise.
Figure 2: Parallel averaging vs sequential correction: the forest’s trees are independent and deep, the boosted trees are shallow and chained through residuals, with a learning rate throttling each contribution.
Mathematical Formulation:
![r_{i,m} = -\left[\frac{\partial\, \ell(y_i, F(x_i))}{\partial\, F(x_i)}\right]_{F = F_{m-1}}](https://s0.wp.com/latex.php?latex=r_%7Bi%2Cm%7D+%3D+-%5Cleft%5B%5Cfrac%7B%5Cpartial%5C%2C+%5Cell%28y_i%2C+F%28x_i%29%29%7D%7B%5Cpartial%5C%2C+F%28x_i%29%7D%5Cright%5D_%7BF+%3D+F_%7Bm-1%7D%7D&bg=ffffff&fg=000&s=3&c=20201002)
Where:
is the ensemble after stage 
, 
the tree added at that stage, and 
the learning rate (shrinkage, typically 0.01-0.1).
is the pseudo-residual for example 
: the negative gradient of the loss 
with respect to the current prediction, i.e. the direction that most reduces the loss.- For squared loss,
, the ordinary residual; other losses (logistic, absolute) change what the trees are fit to.
is the ensemble after stage 
, 
the tree added at that stage, and 
the learning rate (shrinkage, typically 0.01-0.1).
is the pseudo-residual for example 
: the negative gradient of the loss 
with respect to the current prediction, i.e. the direction that most reduces the loss.
, the ordinary residual; other losses (logistic, absolute) change what the trees are fit to.| Aspect | Random Forest | Gradient Boosting |
|---|---|---|
| Training | Independent trees, bootstrap samples, fully parallel | Sequential; each tree fits current pseudo-residuals |
| Tree Shape | Deep, low-bias, high-variance | Shallow (depth 3-8), high-bias learners |
| What It Reduces | Variance (averaging decorrelated trees) | Bias (stage-wise error correction) |
| Noise Robustness | Robust; noise averages out | Fragile; late trees chase label noise |
| Known Issue | Averaging cannot reduce base-tree bias | Target leakage in gradients; fixed by CatBoost ordered boosting |
Leave a Reply