Analysis of a Random Forests Model

Random forests are a scheme proposed by Leo Breiman in the 2000’s for building a predictor ensemble with a set of decision trees that grow in randomly selected subspaces of data. Despite growing interest and practical use, there has been little exploration of the statistical properties of random forests, and little is known about the mathematical forces driving the algorithm. In this paper, we offer an in-depth analysis of a random forests model suggested by Breiman in \cite{Bre04}, which is very close to the original algorithm. We show in particular that the procedure is consistent and adapts to sparsity, in the sense that its rate of convergence depends only on the number of strong features and not on how many noise variables are present.

💡 Research Summary

The paper provides a rigorous statistical analysis of a Random Forests algorithm that closely follows the original scheme introduced by Leo Breiman in 2004. The authors first formalize the algorithm: each tree is grown on a bootstrap sample of the training data, and at each split a random subset of m features (out of the full d features) is considered for partitioning. This random subspace selection reduces correlation among trees and enhances the ensemble effect.

The core theoretical contribution is a proof of consistency. Consistency means that as the number of training observations n tends to infinity, the Random Forest estimator (\hat f_n(x)) converges in probability to the true regression function (f(x)) for every point x. The proof proceeds in two stages. First, the authors show that an individual decision tree is pointwise consistent provided its depth grows sufficiently with n, because each leaf then covers an increasingly small region of the input space and the leaf average converges to the conditional expectation. Second, they demonstrate that averaging a large number B of independent trees eliminates the bias introduced by any single tree and reduces variance at a rate (O(1/B)). Consequently, the ensemble estimator inherits the consistency of its components.

A second major result concerns adaptation to sparsity. In high‑dimensional settings many variables are pure noise, while only a small subset s of “strong” variables actually influences the response. The authors prove that the convergence rate of the Random Forest depends only on s and not on the ambient dimension d. The intuition is that random feature selection gives a higher probability of picking a strong variable at each split; this bias is amplified across the many trees, causing the ensemble to focus on the informative subspace. Formally, the excess risk satisfies a bound of order
\