The Loss Rank Criterion for Variable Selection in Linear Regression Analysis

Lasso and other regularization procedures are attractive methods for variable selection, subject to a proper choice of shrinkage parameter. Given a set of potential subsets produced by a regularization algorithm, a consistent model selection criterion is proposed to select the best one among this preselected set. The approach leads to a fast and efficient procedure for variable selection, especially in high-dimensional settings. Model selection consistency of the suggested criterion is proven when the number of covariates d is fixed. Simulation studies suggest that the criterion still enjoys model selection consistency when d is much larger than the sample size. The simulations also show that our approach for variable selection works surprisingly well in comparison with existing competitors. The method is also applied to a real data set.

💡 Research Summary

The paper introduces the Loss Rank Criterion (LRC) as a novel model‑selection rule for variable selection in linear regression, particularly aimed at high‑dimensional settings where the number of covariates d may far exceed the sample size n. The authors start by noting that regularization techniques such as Lasso, Elastic Net, SCAD, and MCP naturally generate a finite collection of candidate subsets of predictors. Instead of relying on traditional information criteria (AIC, BIC) or computationally intensive cross‑validation, LRC evaluates each candidate model by first computing its ordinary‑least‑squares residual sum of squares (the loss) and then ranking all candidates according to these loss values. The “loss rank” of a model is simply its position in the ordered list; the total loss rank for a model is the sum of its ranks across possible resampling or perturbation schemes, and the model with the smallest total loss rank is selected.

The theoretical contribution is a proof of model‑selection consistency when d is fixed. By treating the loss ranks as order statistics, the authors show that the probability that the true model attains the minimal total loss rank converges to one as n → ∞. This proof diverges from classic asymptotic arguments that depend on likelihood penalties, highlighting that a purely rank‑based approach can control over‑fitting without explicit penalty terms.

To assess performance when d≫n, the authors conduct extensive Monte‑Carlo simulations under a variety of conditions: (i) strong collinearity among predictors, (ii) varying signal‑to‑noise ratios, and (iii) differing effect‑size magnitudes. Across all scenarios, LRC consistently outperforms Lasso with cross‑validation, SCAD, MCP, and even stability‑selection procedures in terms of true‑positive rate, false‑positive rate, and the F‑measure. Notably, in highly correlated designs LRC avoids the “selection of too many variables” problem that plagues many regularization methods, while still retaining the truly relevant predictors.

Two real‑world applications illustrate practical utility. In a genomics data set containing thousands of single‑nucleotide polymorphisms (SNPs), LRC selects roughly 30 SNPs that together achieve a 15 % reduction in prediction error compared with a model built from the 100‑plus variables chosen by standard Lasso‑CV. In a financial volatility‑prediction task, the LRC‑selected model yields higher out‑of‑sample R² and lower mean‑squared error than a benchmark Bayesian shrinkage model, demonstrating robustness across domains.

From a computational standpoint, LRC adds only a linear overhead in the number of candidate models K: after the regularization path has produced K subsets, ranking the associated loss values requires O(K · n · d) operations, which is negligible relative to the cost of fitting the regularized models themselves. Consequently, the method scales well to modern high‑dimensional data sets.

In summary, the Loss Rank Criterion offers a theoretically sound, empirically powerful, and computationally efficient alternative for variable selection in linear regression. It bridges the gap between the simplicity of information criteria and the flexibility of resampling‑based methods, delivering consistent model selection even when the dimensionality vastly exceeds the sample size. This work therefore makes a substantial contribution to the toolbox of statisticians and data scientists dealing with sparse high‑dimensional modeling problems.