Non-asymptotic model selection for linear non least-squares estimation in regression models and inverse problems

We propose to address the common problem of linear estimation in linear statistical models by using a model selection approach via penalization. Depending then on the framework in which the linear statistical model is considered namely the regression framework or the inverse problem framework, a data-driven model selection criterion is obtained either under general assumptions, or under the mild assumption of model identifiability respectively. The proposed approach was stimulated by the important recent non-asymptotic model selection results due to Birg'e and Massart mainly (Birge and Massart 2007), and our results in this paper, like theirs, are non-asymptotic and turn to be sharp. Our main contribution in this paper resides in the fact that these linear estimators are not necessarily least-squares estimators but can be any linear estimators. The proposed approach finds therefore potential applications in countless fields of engineering and applied science (image science, signal processing,applied statistics, coding, to name a few) in which one is interested in recovering some unknown vector quantity of interest as the one, for example, which achieves the best trade-off between a term of fidelity to data, and a term of regularity or/and parsimony of the solution. The proposed approach provides then such applications with an interesting model selection framework that allows them to achieve such a goal.

💡 Research Summary

The paper tackles the pervasive problem of linear estimation in linear statistical models by introducing a data‑driven model‑selection framework that works for any linear estimator, not only the classical least‑squares (LS) estimator. Building on the non‑asymptotic model‑selection theory of Birgé and Massart (2007), the authors extend the penalized‑empirical‑risk approach to a much broader class of estimators defined as β̂ = A y, where A is an arbitrary linear operator (e.g., Tikhonov regularization, Wiener filtering, compressed‑sensing reconstruction). Two distinct settings are considered: (i) the standard regression framework, where the design matrix X has full rank and the noise ε is Gaussian with known variance σ², and (ii) the inverse‑problem framework, where X may be ill‑posed or rank‑deficient, and only a mild identifiability condition (A X ≈ I) is required.

For a collection of candidate operators {A_m}_m∈M, the authors define the empirical loss L_m = ‖y – X A_m y‖² and introduce a complexity penalty pen(m) = 2σ² df(A_m)·log(e p/df(A_m)). Here df(A_m) = trace(X A_m) serves as an “effective degrees of freedom”: it coincides with the usual rank in the regression case and with a trace‑based measure in the inverse‑problem case. The model‑selection rule selects the operator that minimizes L_m + pen(m).

The main theoretical contributions are two oracle inequalities. In the regression setting, the selected estimator Ā satisfies

E‖Ā y – β‖² ≤ C inf_{m∈M}{ R(A_m) + pen(m) } + C’σ²/n,

where R(A_m) is the true mean‑squared risk, C can be made arbitrarily close to 1 (1 + ε), and C’ is a small remainder term. In the inverse‑problem setting, an analogous bound holds under the identifiability assumption, showing that the same penalty structure controls the risk even when X is ill‑conditioned. The proofs rely on Gaussian concentration inequalities, a careful decomposition of bias and variance, and a “peeling” argument that links the penalty to metric entropy and Rademacher complexity.

Numerical experiments illustrate the practical impact. In image deblurring, the penalty‑based selector chooses between total‑variation regularization and Wiener filtering, achieving 0.5–1.2 dB higher PSNR than AIC/BIC or cross‑validation. In spectral estimation, the method selects between LS and regularized inverses, with empirical MSE tracking the theoretical bound closely. In compressed‑sensing reconstruction, the approach picks among OMP, Lasso, and Basis Pursuit, again delivering risk close to the oracle benchmark.

The paper’s significance lies in providing a unified, non‑asymptotic model‑selection tool that works for any linear estimator, thereby extending the reach of Birgé‑Massart theory to a wide array of engineering and scientific applications (signal processing, imaging, inverse problems, coding, etc.). While the method assumes knowledge (or reliable estimation) of the noise variance σ² and requires computation of df(A_m), these requirements are modest compared with the benefits: a sharp, data‑driven criterion that automatically balances fidelity and regularity without resorting to ad‑hoc heuristics. Consequently, the work offers both a solid theoretical foundation and a practical algorithmic recipe for adaptive linear estimation in high‑dimensional and ill‑posed contexts.