Trace Lasso: a trace norm regularization for correlated designs

Using the $\ell_1$-norm to regularize the estimation of the parameter vector of a linear model leads to an unstable estimator when covariates are highly correlated. In this paper, we introduce a new penalty function which takes into account the correlation of the design matrix to stabilize the estimation. This norm, called the trace Lasso, uses the trace norm, which is a convex surrogate of the rank, of the selected covariates as the criterion of model complexity. We analyze the properties of our norm, describe an optimization algorithm based on reweighted least-squares, and illustrate the behavior of this norm on synthetic data, showing that it is more adapted to strong correlations than competing methods such as the elastic net.

💡 Research Summary

The paper introduces the “Trace Lasso,” a novel regularization technique designed to address the instability of the classic Lasso (ℓ₁‑norm) when covariates are highly correlated. The authors observe that existing remedies—ridge regression (ℓ₂), Elastic Net (a convex combination of ℓ₁ and ℓ₂), and Group Lasso—either ignore the precise correlation structure or require prior knowledge of groups. To overcome these limitations, they propose measuring model complexity by the trace (nuclear) norm of the matrix formed by the design matrix X multiplied by a diagonal matrix of the coefficients, i.e., Ω(w)=‖X Diag(w)‖*. This quantity is a convex surrogate for the rank of the selected predictors, which reflects the dimensionality of the subspace spanned by them.

Key theoretical contributions include: (1) Demonstrating that Ω(w) interpolates between the ℓ₁‑norm (when columns of X are orthogonal) and the ℓ₂‑norm (when all columns are identical), thereby automatically adapting to the correlation structure without extra hyper‑parameters. (2) Proving that, under a strongly convex loss, the penalized empirical risk has a unique minimizer, guaranteeing stability. (3) Introducing a broader family of penalties Ω_P(w)=‖P Diag(w)‖* for any matrix P with unit‑norm columns, showing that ℓ₁, ℓ₂, Group Lasso, and the Trace Lasso are special cases. They also establish simple bounds: ‖w‖₂ ≤ Ω_P(w) ≤ ‖w‖₁, and provide an upper bound on the dual norm via the operator norm of P Diag(u).

For optimization, the authors exploit the variational formulation of the nuclear norm: ‖M‖* = ½ inf_{S≻0} tr(MᵀS⁻¹M) + tr(S). By alternating between updating the coefficient vector w and the auxiliary positive‑definite matrix S, they derive an iteratively re‑weighted least‑squares algorithm. The S‑update has a closed form involving the eigen‑decomposition of X Diag(w)²Xᵀ, while the w‑update reduces to solving (XᵀX + λD)w = Xᵀy, where D is a diagonal matrix derived from S⁻¹. This linear system can be solved efficiently with conjugate‑gradient methods, yielding an overall per‑iteration cost of O(np). A decreasing sequence of regularization parameters μ_i ensures numerical stability.

Empirical evaluation on synthetic datasets with block‑diagonal, Toeplitz, and clustered correlation structures compares Trace Lasso against Elastic Net, standard Lasso, and Group Lasso. Results show that Trace Lasso maintains consistent variable selection across repetitions in high‑correlation regimes, while achieving prediction errors comparable to or better than competing methods. Importantly, only a single regularization parameter λ needs to be tuned, unlike Elastic Net’s two‑parameter grid search.

In conclusion, Trace Lasso offers a principled, data‑adaptive regularizer that automatically balances sparsity and stability according to the underlying design matrix correlations. Its unique minimizer, efficient optimization scheme, and superior empirical performance make it a compelling alternative for high‑dimensional regression problems where covariate correlation is pronounced. Future work may explore theoretical generalization bounds, extensions to non‑linear models, and applications to real‑world high‑throughput data such as genomics or imaging.