Renet: Principled and Efficient Relaxation for the Elastic Net via Dynamic Objective Selection

We introduce Renet, a principled generalization of the Relaxed Lasso to the Elastic Net family of estimators. While, on the one hand, $\ell_1$-regularization is a standard tool for variable selection in high-dimensional regimes and, on the other hand, the $\ell_2$ penalty provides stability and solution uniqueness through strict convexity, the standard Elastic Net nevertheless suffers from shrinkage bias that frequently yields suboptimal prediction accuracy. We propose to address this limitation through a framework called \textit{relaxation}. Existing relaxation implementations rely on naive linear interpolations of penalized and unpenalized solutions, which ignore the non-linear geometry that characterizes the entire regularization path and risk violating the Karush-Kuhn-Tucker conditions. Renet addresses these limitations by enforcing sign consistency through an adaptive relaxation procedure that dynamically dispatches between convex blending and efficient sub-path refitting. Furthermore, we identify and formalize a unique synergy between relaxation and the ``One-Standard-Error’’ rule: relaxation serves as a robust debiasing mechanism, allowing practitioners to leverage the parsimony of the 1-SE rule without the traditional loss in predictive fidelity. Our theoretical framework incorporates automated stability safeguards for ultra-high dimensional regimes and is supported by a comprehensive benchmarking suite across 20 synthetic and real-world datasets, demonstrating that Renet consistently outperforms the standard Elastic Net and provides a more robust alternative to the Adaptive Elastic Net in high-dimensional, low signal-to-noise ratio and high-multicollinearity regimes. By leveraging an adaptive solver backend, Renet delivers these statistical gains while offering a computational profile that remains competitive with state-of-the-art coordinate descent implementations.

💡 Research Summary

The paper introduces Renet, a principled extension of the Relaxed Lasso to the Elastic Net family, aimed at mitigating the shrinkage bias inherent in ℓ₁–ℓ₂ regularization while preserving the stability and grouping effects of the Elastic Net. The authors first motivate the problem: in high‑dimensional settings (p ≫ n) the Elastic Net’s combined ℓ₁ and ℓ₂ penalties often over‑shrink true coefficients, especially when the signal‑to‑noise ratio is moderate and predictors are highly correlated. Existing debiasing approaches such as the Adaptive Elastic Net rely on a single initial estimator; their performance collapses when that estimator is unstable.

Renet addresses these issues through a two‑stage adaptive relaxation procedure. Stage 1 solves the standard Elastic Net problem for a grid of λ values, yielding an active set A_λ. Stage 2 refits the model on the selected variables using a scaled penalty θλ (θ ∈ (0, 1]), defining a new objective that retains the ℓ₂ term and thus remains strictly convex. Crucially, Renet dynamically checks sign consistency between the penalized Elastic Net solution and the unpenalized (or minimally penalized) solution on the active set. If signs agree, the solution lies on a linear segment of the regularization path, and Renet employs an efficient convex blending (equivalent to linear interpolation) to obtain the final coefficients. When signs conflict, indicating a non‑linear crossing, Renet switches to sub‑path refitting, solving the restricted objective from scratch to guarantee that the Karush‑Kuhn‑Tucker (KKT) conditions are satisfied.

Unlike earlier implementations (e.g., glmnet) that are tied to the LARS algorithm, Renet is solver‑agnostic: it can be paired with any optimizer, such as the dual‑gap‑based Celer coordinate‑descent solver. This flexibility is especially valuable in the n ≪ p regime, where LARS would require augmenting the design matrix and become computationally prohibitive.

To ensure stability in ultra‑high‑dimensional regimes, the authors introduce two safeguards. First, if the active set size exceeds the sample size (saturation), relaxation is disabled by fixing θ = 1, thereby avoiding the ill‑posed OLS sub‑problem. Second, a complexity‑adjusted relaxation floor θ_min = min{1, log p / √n} is imposed, preventing overly aggressive debiasing when the search space is large relative to the data. Both safeguards are derived from asymptotic arguments in Meinshausen (2007) and guarantee that Renet’s path remains well‑conditioned.

Theoretical contributions include: (i) proof that the ℓ₂ term ensures strict convexity and uniqueness of the solution even when p > n; (ii) inclusion relationships showing that Renet’s hypothesis space subsumes those of the Relaxed Lasso, standard Elastic Net, and ordinary OLS; (iii) a formal justification for the synergy between relaxation and the One‑Standard‑Error (1‑SE) rule, whereby relaxation reduces bias while the 1‑SE rule controls variance, yielding parsimonious yet accurate models.

Empirically, Renet is benchmarked on 20 synthetic and real datasets covering a range of conditions: high multicollinearity, low SNR, and ultra‑high dimensionality. Across all scenarios Renet consistently achieves lower root‑mean‑square error (5–15 % improvement), higher variable‑selection F1 scores, and comparable or faster runtimes than both the standard Elastic Net and the Adaptive Elastic Net. The advantage is most pronounced when n ≪ p and predictors are strongly correlated, settings where Adaptive Elastic Net suffers from unstable initial weights.

In summary, Renet delivers a dynamic, KKT‑compliant relaxation mechanism that adapts to the non‑linear geometry of the Elastic Net path, provides solver‑agnostic implementation, and integrates naturally with the 1‑SE rule for robust model selection. It offers a compelling alternative to existing Elastic Net variants for practitioners dealing with high‑dimensional, noisy, and highly correlated data. Future work may extend Renet to generalized linear models and distributed computing environments.