A Theory of Universal Agnostic Learning

We provide a complete theory of optimal universal rates for binary classification in the agnostic setting. This extends the realizable-case theory of Bousquet, Hanneke, Moran, van Handel, and Yehudayoff (2021) by removing the realizability assumption on the distribution. We identify a fundamental tetrachotomy of optimal rates: for every concept class, the optimal universal rate of convergence of the excess error rate is one of $e^{-n}$, $e^{-o(n)}$, $o(n^{-1/2})$, or arbitrarily slow. We further identify simple combinatorial structures which determine which of these categories any given concept class falls into.

💡 Research Summary

This paper develops a comprehensive theory of optimal universal learning rates for binary classification in the agnostic (non‑realizable) setting. While classical statistical learning theory, based on VC dimension, provides uniform convergence guarantees of order n⁻¹/² (or a constant lower bound for infinite VC dimension), such uniform rates are often overly pessimistic for a fixed data distribution. To address this, the authors introduce the notion of a universal learning rate: a sequence R(n)→0 is achievable if there exists a learning algorithm whose expected excess risk E