ML0087 AdaBoost vs Gradient Boosting

AdaBoost and gradient boosting are both sequential ensemble methods that add weak learners stage by stage, but they differ in what each new learner targets and how the loss is shaped. AdaBoost reweights training examples: misclassified samples get exponentially higher weights, and the next weak learner is fit on the reweighted data, then added with a stage weight set by the log-odds of its weighted error rate. Gradient boosting fits each new learner to the negative gradient of an arbitrary differentiable loss with respect to the current ensemble’s predictions, which generalizes to regression, ranking, and any custom loss. AdaBoost’s exponential loss is a special, fixed case; gradient boosting’s loss is a knob. In production, gradient boosting variants (XGBoost, LightGBM, CatBoost) dominate tabular ML, while AdaBoost is largely legacy, and scikit-learn even removed the SAMME.R variant in 2024.

(1) Reweighting vs Gradient Fitting: AdaBoost changes sample weights via the exponential loss and refits on the weighted distribution; gradient boosting fits each tree to pseudo-residuals, the negative gradient of whatever loss you choose.
(2) Fixed vs Arbitrary Loss: AdaBoost’s reweighting rule is tied to the exponential loss (SAMME for multi-class), so a different task needs a different algorithm (regression uses AdaBoost.R2); gradient boosting swaps in squared error, logistic loss, quantile loss, Huber, or a custom loss without changing the algorithm, covering regression, classification, and ranking.
(3) Production Dominance: gradient boosting frameworks (XGBoost, LightGBM, CatBoost) dominate tabular ML in production, while AdaBoost appears only in niche ensembles; scikit-learn removed SAMME.R in version 1.6 (deprecated one release earlier) because the algorithm was based on a preprint never published in its final form.

Figure 1: AdaBoost updates sample weights via the exponential loss and refits on the reweighted distribution, while gradient boosting computes pseudo-residuals as the negative gradient of an arbitrary loss and fits the next tree to those residuals.

The exponential loss also makes AdaBoost fragile in ways gradient boosting is not. Because misclassified samples receive exponentially growing weights, a single noisy label or outlier can dominate subsequent stages, pulling the ensemble toward the noise. Gradient boosting’s gradient-based approach spreads the influence of an outlier according to the loss derivative, and modern frameworks add regularization (shrinkage, tree depth limits, column subsampling, early stopping) that AdaBoost lacks natively. A 2024 NeurIPS paper proved that boosting can in principle optimize essentially any loss, without requiring convexity, differentiability, or even continuity, but this remains theoretical; in practice, gradient boosting’s differentiable-loss flexibility plus regularization is what made XGBoost, LightGBM, and CatBoost the de facto tabular standard.

Mathematical Formulation:

Where:

• is the AdaBoost weight of example at stage , is the stage weight (half the log-odds of being correct), and normalizes the weights so they sum to 1.
• is the gradient boosting ensemble after stage , is the tree fit to the pseudo-residuals, and is the learning rate (shrinkage).
• is the pseudo-residual: the negative gradient of the chosen loss with respect to the current prediction. For squared loss this reduces to the ordinary residual ; for logistic loss it is .