Taking Advantage of Sparsity in Multi-Task Learning

We study the problem of estimating multiple linear regression equations for the purpose of both prediction and variable selection. Following recent work on multi-task learning Argyriou et al. [2008], we assume that the regression vectors share the same sparsity pattern. This means that the set of relevant predictor variables is the same across the different equations. This assumption leads us to consider the Group Lasso as a candidate estimation method. We show that this estimator enjoys nice sparsity oracle inequalities and variable selection properties. The results hold under a certain restricted eigenvalue condition and a coherence condition on the design matrix, which naturally extend recent work in Bickel et al. [2007], Lounici [2008]. In particular, in the multi-task learning scenario, in which the number of tasks can grow, we are able to remove completely the effect of the number of predictor variables in the bounds. Finally, we show how our results can be extended to more general noise distributions, of which we only require the variance to be finite.

💡 Research Summary

The paper addresses the simultaneous estimation of several linear regression models in a multi‑task learning setting, with the dual goals of accurate prediction and reliable variable selection. The authors adopt the realistic assumption that all tasks share the same sparsity pattern: the set of truly relevant predictors is identical across tasks, while all other predictors are irrelevant for every task. To exploit this structure they propose using the Group Lasso, a regularization method that groups together the coefficients of each predictor across all tasks and applies an ℓ1 penalty to the ℓ2‑norm of each group. This encourages entire groups (i.e., predictors) to be set to zero, thereby enforcing a common support.

The theoretical analysis rests on two high‑dimensional conditions that extend those used for the ordinary Lasso. First, a Restricted Eigenvalue (RE) condition on the design matrix guarantees that, for any matrix of coefficient differences Δ satisfying a cone constraint (‖Δ_{S^c}‖{2,1} ≤ 3‖Δ_S‖{2,1}), the empirical loss is strongly convex on the support S. Formally, (1/n)‖XΔ‖_F² ≥ κ‖Δ_S‖_F² for some κ>0. Second, a coherence condition limits the pairwise correlation between distinct columns of X, preventing excessive multicollinearity. Together these assumptions allow the authors to derive sharp non‑asymptotic bounds.

The first major result is a sparsity oracle inequality for the Group Lasso estimator (\hat B). Under the RE and coherence conditions and with an appropriately chosen regularization parameter λ, the prediction error satisfies

\