Adaptive Ridge Selector (ARiS)

We introduce a new shrinkage variable selection operator for linear models which we term the \emph{adaptive ridge selector} (ARiS). This approach is inspired by the \emph{relevance vector machine} (RVM), which uses a Bayesian hierarchical linear setup to do variable selection and model estimation. Extending the RVM algorithm, we include a proper prior distribution for the precisions of the regression coefficients, $v_{j}^{-1} \sim f(v_{j}^{-1}|\eta)$, where $\eta$ is a scalar hyperparameter. A novel fitting approach which utilizes the full set of posterior conditional distributions is applied to maximize the joint posterior distribution $p(\boldsymbol\beta,\sigma^{2},\mathbf{v}^{-1}|\mathbf{y},\eta)$ given the value of the hyper-parameter $\eta$. An empirical Bayes method is proposed for choosing $\eta$. This approach is contrasted with other regularized least squares estimators including the lasso, its variants, nonnegative garrote and ordinary ridge regression. Performance differences are explored for various simulated data examples. Results indicate superior prediction and model selection accuracy under sparse setups and drastic improvement in accuracy of model choice with increasing sample size.

💡 Research Summary

The paper introduces the Adaptive Ridge Selector (ARiS), a novel Bayesian variable‑selection and shrinkage method for linear regression. Building on the hierarchical formulation of the Relevance Vector Machine (RVM), ARiS places a normal prior on the regression coefficients β conditional on the error variance σ² and a diagonal precision matrix V = diag(v₁,…,v_p). Each precision element v_j⁻¹ follows an independent Gamma distribution with shape η + 1 and rate μ, where η is a scalar hyper‑parameter controlling the amount of shrinkage and μ is a small scale constant.

From the likelihood y ∼ N(Xβ,σ²I) and the priors, the joint posterior p(β,σ²,v⁻¹|y,η,μ) is derived. The conditional posterior distributions are analytically tractable: β|σ²,v⁻¹,y is multivariate normal, σ²|β,v⁻¹,y is inverse‑Gamma, and each v_j⁻¹|β_j,σ²,y is Gamma. By iteratively maximizing these conditionals (the Lindley‑Smith algorithm), the method reduces at each iteration to a weighted ridge regression:

β^{(l)} = arg min_β ‖y − Xβ‖² + (1 + 2η) ∑_j β_j² ω_j^{(l)},

where ω_j^{(l)} = √{β_j^{(l‑1)2}/σ²^{(l)}}. This re‑weighting scheme shrinks coefficients toward zero in proportion to their current magnitude and the residual variance, yielding a sparse solution when η is positive. The algorithm starts from the ordinary least‑squares estimate, updates the weights ω_j using the expected values of v_j⁻¹, solves the ridge sub‑problem, and then updates σ²; the process repeats until convergence.

The hyper‑parameter η is selected by an empirical Bayes (EB) approach that maximizes the marginal likelihood p(y|η). Because the marginal likelihood is analytically intractable, the authors propose two approximations: a Laplace approximation based on a second‑order Taylor expansion around the posterior mode, and a Monte‑Carlo integration using Gibbs sampling of the conditional posteriors. Both approximations give similar η̂ values, with the Laplace method being computationally cheaper.

Theoretical analysis shows that the marginal prior for β induced by the hierarchical Gamma‑Gamma construction is a product of univariate Student‑t densities, producing “ridges” along each coordinate axis. Larger η values increase the effective ℓ₂ penalty (ridge‑like behavior), while η → 0 approaches an exponential prior (ℓ₁‑like behavior). Thus ARiS interpolates continuously between ridge, lasso, and RVM regimes, offering a unified framework.

Extensive simulations explore a range of sparsity levels, sample sizes (n = 50–400), and numbers of predictors (p = 100–500). Performance metrics include mean squared prediction error (MSE), variable‑selection precision/recall/F₁, and model size. Across all settings, ARiS achieves lower MSE than lasso, elastic‑net, non‑negative garrote, and ordinary ridge regression, with especially pronounced gains when n ≫ p. Variable‑selection accuracy is also superior: ARiS maintains high precision while recovering most true signals, leading to fewer false positives than competing methods. As n grows, the EB estimate of η stabilizes, and the selected model converges to the true sparse set.

A real‑data illustration (e.g., a gene‑expression or sports‑performance dataset) confirms the simulation findings: ARiS produces a parsimonious model with strong predictive performance and interpretable selected variables.

In summary, ARiS leverages a fully Bayesian hierarchical model to obtain closed‑form conditional posteriors, enabling an efficient deterministic optimization that avoids costly MCMC. The empirical Bayes selection of η removes the need for manual tuning of penalty parameters, while the re‑weighting interpretation connects the method to classic regularization techniques. The authors suggest future extensions to kernelized (non‑linear) models and multivariate response settings, indicating the broad applicability of the ARiS framework.