ML0064 Random Forest II

Written by

Please explain the benefits and drawbacks of random forest.

Answer

Random forest is a powerful ensemble method that reduces overfitting and improves predictive accuracy by combining many decision trees. The trade-off: it sacrifices interpretability and computational efficiency, and it may require careful tuning on large, imbalanced, or sparse datasets.

(1) Benefit: Reduces Overfitting: Aggregating many trees lowers variance.
(2) Benefit: Robust: Less sensitive to noise and outliers; handles high-dimensional data well.
(3) Benefit: Feature Importance: Built-in estimates identify influential variables; bagging improves generalization.
(4) Drawback: Cost: Hard to interpret compared to a single tree; slower to train and predict; large forests consume significant memory.
(5) Drawback: Data Traps: Class imbalance can bias predictions, and very sparse data can make it underperform other algorithms.

Random forest versus logistic regression on an imbalanced dataset with minority F1 scores in the titles

Figure 1: The imbalance trap: on 96/4 skewed data the random forest’s regions bend toward the majority class and its minority-class F1 (0.73) loses even to a class-weighted logistic regression (0.76): a strong default is not automatically strong everywhere.

Mathematical Formulation:
$\mathrm{Var}\big(\bar{T}(x)\big) = \rho \, \sigma^2 + \frac{1 - \rho}{B} \, \sigma^2$

Where:

$\bar{T}(x)$ is the forest’s averaged prediction over $B$ trees, each with individual variance $\sigma^2$ .
$\rho$ is the pairwise correlation between trees: this is what bagging and random feature subsets try to shrink.
As $B \to \infty$ the second term vanishes but the $\rho\sigma^2$ floor remains: correlation, not tree count, is the limiting factor.