On the Degrees of Freedom of some Lasso procedures

The effective degrees of freedom of penalized regression models quantify the actual amount of information used to generate predictions, playing a pivotal role in model evaluation and selection. Although a closed-form estimator is available for the Lasso penalty, adaptive extensions of widely used penalized approaches, including the Adaptive Lasso and Adaptive Group Lasso, have remained without analogous theoretical characterization. This paper presents the first unbiased estimator of the effective degrees of freedom for these methods, along with their main theoretical properties, for both orthogonal and non-orthogonal designs, derived within Stein’s unbiased risk estimation framework. The resulting expressions feature inflation terms influenced by the regularization parameter, coefficient signs, and least-squares estimates. These advances enable more accurate model selection criteria and unbiased prediction error estimates, illustrated through synthetic and real data. These contributions offer a rigorous theoretical foundation for understanding model complexity in adaptive regression, bridging a critical gap between theory and practice.

💡 Research Summary

This paper tackles a fundamental yet under‑explored problem in penalized regression: how to quantify the effective degrees of freedom (df) of adaptive extensions of the Lasso. While the ordinary Lasso enjoys a simple df estimator—the size of the active set—adaptive methods such as the Adaptive Lasso and Adaptive Group Lasso introduce data‑dependent weights that break this simplicity. The authors fill this gap by deriving the first unbiased df estimators for these procedures, using Stein’s unbiased risk estimation (SURE) framework.

The theoretical development proceeds as follows. Starting from Stein’s Lemma, df is expressed as the expected trace of the Jacobian of the fitted values with respect to the response. For the Adaptive Lasso, the KKT conditions are differentiated, revealing an extra term that involves the derivative of the weight function (w_j(|\hat\beta^{LS}_j|)). In an orthonormal design this yields the closed‑form \