Data-driven calibration of linear estimators with minimal penalties

This paper tackles the problem of selecting among several linear estimators in non-parametric regression; this includes model selection for linear regression, the choice of a regularization parameter in kernel ridge regression, spline smoothing or locally weighted regression, and the choice of a kernel in multiple kernel learning. We propose a new algorithm which first estimates consistently the variance of the noise, based upon the concept of minimal penalty, which was previously introduced in the context of model selection. Then, plugging our variance estimate in Mallows’ $C_L$ penalty is proved to lead to an algorithm satisfying an oracle inequality. Simulation experiments with kernel ridge regression and multiple kernel learning show that the proposed algorithm often improves significantly existing calibration procedures such as generalized cross-validation.

💡 Research Summary

The paper addresses the fundamental problem of selecting the best linear estimator among a family of candidates in non‑parametric regression, a setting that includes ordinary linear regression, spline smoothing, kernel ridge regression (KRR), locally weighted regression, and multiple kernel learning (MKL). Traditional selection tools such as AIC, BIC, cross‑validation (CV), and generalized cross‑validation (GCV) either require costly resampling or suffer from instability when the model space is large or the noise level is high. The authors propose a novel, fully data‑driven calibration procedure that hinges on the concept of a “minimal penalty,” originally introduced for model selection, to first obtain a consistent estimate of the noise variance σ², and then plug this estimate into Mallows’ C_L penalty.

The key theoretical insight is that for any linear estimator defined by a smoothing matrix S(λ) (where λ denotes a regularization or kernel‑mixing parameter), the expected risk can be decomposed into a bias term and a variance term proportional to σ²·tr(S(λ)²). Mallows’ C_L = ‖Y−Ŷ(λ)‖² + 2σ²·df(λ) (with df(λ)=tr(S(λ))) is known to be an unbiased estimator of the risk when σ² is known. The difficulty lies in estimating σ² without external information. By examining the empirical relationship between the residual sum of squares ‖Y−Ŷ(λ)‖² and the effective degrees of freedom df(λ) across a grid of λ values, the authors identify a regime where this relationship is essentially linear. The slope of this linear segment provides a consistent estimator (\hatσ²). This “minimal‑penalty” estimator requires no tuning parameters and leverages only the observed data.

With (\hatσ²) in hand, the authors define a data‑driven Mallows’ criterion (\hat C_L(λ)=‖Y−Ŷ(λ)‖² + 2\hatσ²·df(λ)). They prove that the λ minimizing (\hat C_L) satisfies an oracle inequality of the form E