Concentration inequalities of the cross-validation estimate for stable predictors

In this article, we derive concentration inequalities for the cross-validation estimate of the generalization error for stable predictors in the context of risk assessment. The notion of stability has been first introduced by \cite{DEWA79} and extended by \cite{KEA95}, \cite{BE01} and \cite{KUNIY02} to characterize class of predictors with infinite VC dimension. In particular, this covers $k$-nearest neighbors rules, bayesian algorithm (\cite{KEA95}), boosting,… General loss functions and class of predictors are considered. We use the formalism introduced by \cite{DUD03} to cover a large variety of cross-validation procedures including leave-one-out cross-validation, $k$-fold cross-validation, hold-out cross-validation (or split sample), and the leave-$\upsilon$-out cross-validation. In particular, we give a simple rule on how to choose the cross-validation, depending on the stability of the class of predictors. In the special case of uniform stability, an interesting consequence is that the number of elements in the test set is not required to grow to infinity for the consistency of the cross-validation procedure. In this special case, the particular interest of leave-one-out cross-validation is emphasized.

💡 Research Summary

The paper establishes rigorous concentration inequalities for cross‑validation (CV) estimates of the generalization error of learning algorithms that satisfy various notions of stability. Building on the stability concepts introduced by Devroye, Wagner, and later refined by Kearns, Blum, and others, the authors define a unified stability parameter β that quantifies how much the predictor’s loss changes when a single training example is replaced. This framework encompasses pointwise, mean, and uniform stability, thereby covering algorithms with infinite VC dimension such as k‑nearest‑neighbors, Bayesian classifiers, and boosting methods.

To treat a wide range of CV procedures in a single mathematical setting, the paper adopts the “sampling mask” formalism of Dudley (2003). A mask is an n‑by‑n binary matrix that simultaneously specifies the training and test subsets for each CV split. Leave‑one‑out (LOO), k‑fold, hold‑out, and leave‑ν‑out are all special cases of particular mask distributions. This abstraction allows the authors to write the CV estimator (\hat R_{CV}) as an expectation over the mask and to analyze its deviation from the true risk R using the same probabilistic tools.

The core theoretical contribution consists of two theorems. The first theorem shows that for any β‑stable algorithm and any mask with test‑fraction τ (i.e., the proportion of examples placed in the test set), the deviation satisfies a sub‑Gaussian tail bound:

\