ML0063 Random Forest

Random Forest builds an ensemble of decision trees, each trained on a bootstrapped sample of the data with a random feature subset considered at each split. This combination reduces variance, combats overfitting, and improves predictive accuracy; the final output aggregates all trees’ predictions: majority vote for classification, averaging for regression.

(1) Bootstrap Sampling: Create multiple subsets of the training data by sampling with replacement (bootstrap samples).
(2) Grow Decision Trees: Train an unpruned decision tree on each bootstrap sample.
(3) Random Feature Selection: At every split in every tree, consider only a random subset of features; this increases diversity between trees.
(4) Aggregate: Classification: each tree votes for a class and the majority wins; regression: the tree outputs are averaged.

Figure 1: Three trees, three different jagged boundaries: each overfits its own bootstrap sample in its own way. The ensemble’s boundary (bottom right) averages the votes and lands smoother and closer to the true structure: the trees’ individual errors cancel.

Mathematical Formulation:

Where:

• is the prediction of the -th tree for input .
• is the total number of trees in the forest.
• First line: classification by majority vote (mode); second line: regression by averaging.

Figure 2: The full pipeline: B bootstrap replicas of the training set feed B independently grown trees (each with random feature subsets at its splits), and a majority-vote / averaging box fuses their outputs into one robust prediction.