Robustness and Generalization

We derive generalization bounds for learning algorithms based on their robustness: the property that if a testing sample is “similar” to a training sample, then the testing error is close to the training error. This provides a novel approach, different from the complexity or stability arguments, to study generalization of learning algorithms. We further show that a weak notion of robustness is both sufficient and necessary for generalizability, which implies that robustness is a fundamental property for learning algorithms to work.

💡 Research Summary

The paper introduces a novel framework for analyzing the generalization ability of learning algorithms based on robustness, a property that links the similarity of test and training samples to the closeness of their losses. Traditional generalization theory relies on capacity measures such as VC dimension, Rademacher complexity, or algorithmic stability, which often ignore the geometric structure of the data and the algorithm’s sensitivity to input perturbations. By formalizing robustness, the authors provide an alternative route that directly incorporates these aspects.

Definition of robustness.
Let ((\mathcal X, d)) be the input space equipped with a metric (d). For a tolerance (\epsilon>0), two examples (z) and (z’) are called (\epsilon)-neighbors if (d(z,z’)\le\epsilon). An algorithm (A) is said to be ((\epsilon,\gamma(\epsilon)))-robust if for every pair of (\epsilon)-neighbors the loss satisfies (|\ell(A,z)-\ell(A,z’)|\le\gamma(\epsilon)), where (\gamma) is a non‑decreasing function of (\epsilon). Small (\gamma(\epsilon)) for small (\epsilon) indicates strong robustness.

Robustness‑based generalization bound.
Assuming bounded loss in (