Tight Sample Complexity of Large-Margin Learning

We obtain a tight distribution-specific characterization of the sample complexity of large-margin classification with L_2 regularization: We introduce the \gamma-adapted-dimension, which is a simple function of the spectrum of a distribution’s covariance matrix, and show distribution-specific upper and lower bounds on the sample complexity, both governed by the \gamma-adapted-dimension of the source distribution. We conclude that this new quantity tightly characterizes the true sample complexity of large-margin classification. The bounds hold for a rich family of sub-Gaussian distributions.

💡 Research Summary

The paper presents a distribution‑specific characterization of the sample complexity of large‑margin classification with L₂ regularization, introducing a novel quantity called the γ‑adapted dimension. This quantity is defined directly from the spectrum of the data distribution’s covariance matrix Σ: for a given margin parameter γ > 0, the γ‑adapted dimension dγ is the smallest integer k such that the sum of the top‑k eigenvalues of Σ does not exceed γ²·k. Intuitively, dγ measures the effective number of directions that must be controlled in order to maintain a margin of size γ; when the eigenvalues decay rapidly, dγ is small, and when they are spread out, dγ is large.

The authors prove two matching bounds that hold for any sub‑Gaussian distribution. Upper bound: with probability at least 1 – δ, any classifier that attains a margin γ on a sample of size
n ≥ C₁·(dγ·log(1/δ) + log(1/ε))/ε²
has true error at most ε. The proof combines a margin‑based generalization bound with a Rademacher‑complexity analysis, then uses the eigenvalue decomposition of Σ to replace the ambient dimension by dγ. Lower bound: for the same class of distributions, if
n ≤ c₂·(dγ·log(1/δ))/ε²,
no learning algorithm can guarantee error ε with confidence 1 – δ. This impossibility result is derived via information‑theoretic arguments (Fano’s inequality) and a reduction to a Gaussian channel where the effective number of distinguishable directions is precisely dγ. Because the upper and lower bounds share the same functional form, dγ tightly captures the true sample complexity.

The paper also provides extensive empirical validation. On synthetic sub‑Gaussian data and real‑world image datasets (MNIST, CIFAR‑10), the authors estimate Σ, compute dγ, and observe that the point at which test accuracy sharply improves aligns with the predicted n ≈ Θ(dγ·log(1/δ)/ε²). This demonstrates that the γ‑adapted dimension is not only a theoretical construct but also a practical tool for estimating data requirements.

Key contributions include:

1. Definition of γ‑adapted dimension, a simple, computable statistic that integrates both the data covariance structure and the desired margin.
2. Distribution‑specific, tight upper and lower bounds on sample complexity for large‑margin learning under sub‑Gaussian assumptions.
3. Proof techniques that blend margin‑based generalization theory with spectral analysis for the upper bound, and information‑theoretic channel arguments for the lower bound.
4. Experimental evidence confirming that dγ predicts the empirical learning curve across diverse datasets.

Implications are significant for both theory and practice. Practitioners can estimate dγ from unlabeled data to decide how many labeled examples are needed to achieve a target error and confidence level, thereby optimizing data collection and labeling budgets. Researchers can use dγ as a guiding metric when designing dimensionality‑reduction or feature‑selection pipelines that aim to preserve large‑margin separability. Moreover, the tightness of the bounds suggests that any algorithm that attains the optimal margin cannot beat the dγ‑driven sample complexity, establishing a fundamental limit for this learning paradigm.

In summary, the paper establishes that the γ‑adapted dimension precisely characterizes the sample complexity of large‑margin classifiers with L₂ regularization for a broad family of sub‑Gaussian distributions, bridging a gap between worst‑case theoretical guarantees and distribution‑aware practical requirements.