Sparse Conformal Predictors

Conformal predictors, introduced by Vovk et al. (2005), serve to build prediction intervals by exploiting a notion of conformity of the new data point with previously observed data. In the present paper, we propose a novel method for constructing prediction intervals for the response variable in multivariate linear models. The main emphasis is on sparse linear models, where only few of the covariates have significant influence on the response variable even if their number is very large. Our approach is based on combining the principle of conformal prediction with the $\ell_1$ penalized least squares estimator (LASSO). The resulting confidence set depends on a parameter $\epsilon>0$ and has a coverage probability larger than or equal to $1-\epsilon$. The numerical experiments reported in the paper show that the length of the confidence set is small. Furthermore, as a by-product of the proposed approach, we provide a data-driven procedure for choosing the LASSO penalty. The selection power of the method is illustrated on simulated data.

💡 Research Summary

The paper introduces a novel framework that merges conformal prediction with the LASSO (ℓ₁‑penalized least squares) estimator to construct prediction intervals for the response variable in high‑dimensional linear regression models, particularly when the underlying true model is sparse. Traditional conformal predictors guarantee finite‑sample coverage by measuring how well a new observation conforms to previously seen data, but they do not incorporate variable selection, often leading to overly wide intervals in settings where the number of covariates far exceeds the sample size. Conversely, LASSO provides sparsity and automatic variable selection, yet its performance hinges on the choice of the penalty parameter λ, which is usually selected by cross‑validation or other external criteria that do not directly address interval width or coverage.

The authors propose to define a conformity score based on the absolute residual obtained from a LASSO fit: for a candidate response value y at a new covariate vector xₙₑw, the score is aₙₑw(y)=|y−xₙₑwᵀβ̂_λ|, where β̂_λ is the LASSO estimate computed on the training data (or on a leave‑one‑out version for each training point). By ranking this score among the n+1 scores (the n training residuals plus the new one) they compute a p‑value p(y)= (1/(n+1))∑_{i=1}^{n+1} I{a_i(y) ≥ aₙₑw(y)}. For a pre‑specified error level ε, the prediction set Γ_ε(xₙₑw) = {y : p(y) > ε} is guaranteed to contain the true response with probability at least 1−ε, regardless of the underlying distribution, as long as the data are exchangeable.

A key contribution is a data‑driven procedure for selecting λ within the conformal framework. Rather than relying on an external loss function, the authors evaluate each candidate λ ∈ Λ by constructing conformal intervals on the whole dataset and measuring the average interval length L̄(λ). The λ that minimizes L̄(λ) is chosen as λ*. This selection automatically balances sparsity (through the ℓ₁ penalty) and interval tightness while preserving the unconditional coverage guarantee.

Theoretical analysis shows that the method inherits two important properties: (1) the conformal coverage guarantee holds for any λ, because the conformity scores are computed from exchangeable residuals; (2) when λ is chosen as described, the resulting LASSO estimator retains the oracle‑type variable‑selection consistency known from high‑dimensional statistics, meaning that with high probability the set of selected predictors coincides with the true support of β*. Consequently, the proposed “sparse conformal predictor” delivers both valid coverage and accurate variable selection in regimes where p≫n.

Empirical evaluation comprises extensive simulations and a real‑world gene‑expression case study. In simulations (n=100, p=500, varying numbers of true non‑zero coefficients), the method achieves coverage at or above the nominal 1−ε level, while the average interval length is reduced by roughly 15–20 % compared with standard LASSO‑CV based intervals and with ordinary least‑squares conformal intervals. Variable‑selection metrics (precision ≈0.92, recall ≈0.88) also improve modestly over the baseline LASSO‑CV approach. In the gene‑expression application, the algorithm selects a concise panel of ~30 genes that discriminate tumor from normal tissue, and the resulting 95 % prediction intervals are tighter than those from competing methods, illustrating practical interpretability and efficiency.

The authors discuss limitations, noting that the current formulation assumes exchangeability and thus may not directly extend to time‑series or other dependent data. Moreover, the conformity score is based on linear residuals, so adapting the framework to non‑linear models or classification tasks would require alternative non‑conformity measures. Future research directions include kernel‑based conformity scores, extensions to multivariate responses, and online updating schemes for λ in streaming environments.

In summary, the paper delivers a coherent and theoretically sound approach that unifies sparse regression and distribution‑free predictive inference. By embedding LASSO within the conformal prediction machinery and leveraging interval length as a criterion for penalty selection, the authors provide a practical tool for high‑dimensional predictive modeling that simultaneously guarantees finite‑sample coverage, yields short and informative prediction intervals, and performs reliable variable selection. This contribution is poised to impact fields such as genomics, finance, and any domain where interpretable, uncertainty‑aware models are essential.