ML0065 Random Forest III

Select the number of features considered at each split (the , or max_features) by starting from the default heuristics, then tuning with cross-validation or out-of-bag (OOB) error to find the best value for your specific dataset. The choice trades bias against variance and accuracy against training cost.

(1) Default Heuristics: Classification: ; regression: ; solid starting points.
(2) Bias-Variance Trade-Off: Smaller adds randomness: less correlated trees (lower variance) but potentially higher bias; larger strengthens each tree (lower bias) but correlates their errors (higher variance).
(3) Systematic Search: Grid or randomized search over a range of values with cross-validation is the most robust method; OOB error offers a validation-free alternative unique to bagged models.

Figure 1: CV accuracy across max_features on a 30-feature dataset: accuracy is poor when m is tiny (trees too weak), then plateaus with a noisy peak at m = 15; the heuristic lands inside the good region, and CV refinement picks the best value on the plateau.

Mathematical Formulation:

Where:

• is the total number of features in the dataset.
• is the number of features randomly drawn and considered at each split (max_features in most libraries).
• These are heuristics, not optima; cross-validation or OOB error refines them per dataset.