Harmful Overfitting in Sobolev Spaces

Motivated by recent work on benign overfitting in overparameterized machine learning, we study the generalization behavior of functions in Sobolev spaces $W^{k, p}(\mathbb{R}^d)$ that perfectly fit a noisy training data set. Under assumptions of label noise and sufficient regularity in the data distribution, we show that approximately norm-minimizing interpolators, which are canonical solutions selected by smoothness bias, exhibit harmful overfitting: even as the training sample size $n \to \infty$, the generalization error remains bounded below by a positive constant with high probability. Our results hold for arbitrary values of $p \in [1, \infty)$, in contrast to prior results studying the Hilbert space case ($p = 2$) using kernel methods. Our proof uses a geometric argument which identifies harmful neighborhoods of the training data using Sobolev inequalities.

💡 Research Summary

The paper investigates the generalization behavior of interpolating functions drawn from Sobolev spaces (W^{k,p}(\mathbb{R}^d)) when they perfectly fit noisy training data. While recent work on “benign overfitting” has shown that overparameterized models can interpolate noisy labels yet still generalize well—especially in high‑dimensional linear or kernel settings—this work asks whether the same phenomenon can occur for fixed‑dimension data and for Sobolev spaces beyond the Hilbert case (p=2).

The authors consider a probabilistic data model: inputs (x) lie in a bounded open set (\Omega\subset\mathbb{R}^d) with a density bounded away from zero and infinity; labels (y) are generated conditionally on (x) with a sub‑Gaussian tail and a non‑trivial amount of label noise (there is a fixed probability (\rho>0) that the conditional loss exceeds a positive constant (\sigma)). The loss (\ell) is continuous, zero only when predictions match the true label, and satisfies a mild exponential growth bound, which covers all (\ell_q) losses for (q\in