Fast Overlapping Group Lasso

The group Lasso is an extension of the Lasso for feature selection on (predefined) non-overlapping groups of features. The non-overlapping group structure limits its applicability in practice. There have been several recent attempts to study a more general formulation, where groups of features are given, potentially with overlaps between the groups. The resulting optimization is, however, much more challenging to solve due to the group overlaps. In this paper, we consider the efficient optimization of the overlapping group Lasso penalized problem. We reveal several key properties of the proximal operator associated with the overlapping group Lasso, and compute the proximal operator by solving the smooth and convex dual problem, which allows the use of the gradient descent type of algorithms for the optimization. We have performed empirical evaluations using the breast cancer gene expression data set, which consists of 8,141 genes organized into (overlapping) gene sets. Experimental results demonstrate the efficiency and effectiveness of the proposed algorithm.

💡 Research Summary

The paper tackles a fundamental limitation of the traditional Group Lasso, namely its requirement that feature groups be non‑overlapping. In many real‑world applications—such as genomics, image analysis, or text mining—features naturally belong to multiple groups, and forcing a disjoint partition either discards valuable prior knowledge or leads to sub‑optimal models. The authors therefore focus on the Overlapping Group Lasso (OGL), which extends the Group Lasso penalty to arbitrary collections of possibly intersecting groups, and they develop a highly efficient algorithm for solving the resulting convex optimization problem.

The key technical contribution is a novel treatment of the proximal operator associated with the OGL regularizer. Instead of attempting to compute this operator directly—a task that becomes combinatorially expensive when groups overlap—the authors derive its dual formulation. By introducing auxiliary variables for each group and appropriate weighting matrices, they rewrite the OGL penalty as a constrained problem whose Lagrangian yields a smooth, convex dual objective. This dual problem has a dimensionality comparable to the original feature space and possesses a Lipschitz‑continuous gradient, which makes it amenable to standard first‑order methods. The paper details how accelerated gradient descent, Nesterov’s momentum, or limited‑memory BFGS can be applied to the dual, and how the primal solution is recovered by a simple linear mapping from the optimal dual variables.

From a computational standpoint, the proposed approach dramatically reduces both memory footprint and per‑iteration cost. Traditional methods such as ADMM or sub‑gradient schemes must maintain multiple copies of overlapping variables and perform costly projection steps, leading to O(N·G) complexity where N is the number of features and G the number of groups. In contrast, the dual‑based algorithm scales linearly (or near‑linearly) with the number of features and exploits the inherent sparsity of the group structure, resulting in a per‑iteration complexity of O(N + |E|), where |E| denotes the total number of non‑zero entries in the group‑membership matrix. The authors also prove global convergence under standard strong convexity assumptions and provide explicit bounds on the Lipschitz constant of the dual gradient.

Empirical validation is performed on a large‑scale breast‑cancer gene‑expression dataset comprising 8,141 genes organized into roughly 1,200 overlapping gene sets. The authors compare their method against a state‑of‑the‑art ADMM implementation for OGL and a baseline non‑overlapping Group Lasso that treats each gene set independently. Results show that the new algorithm reaches convergence 3–5 times faster while achieving comparable—or slightly better—predictive performance measured by area under the ROC curve (AUC). Moreover, the selected gene sets align well with known biological pathways, as confirmed by Gene Ontology enrichment analysis, demonstrating that the method preserves interpretability despite its computational shortcuts.

Beyond the presented experiments, the paper outlines several promising extensions. The dual formulation is agnostic to the loss function, allowing straightforward integration with logistic regression, support vector machines, or other convex losses. The authors suggest adapting the algorithm to online or streaming settings by employing incremental gradient updates on the dual. Finally, they discuss generalizing the framework to more complex structured regularizers, such as hierarchical trees or graph‑based overlaps, which could further broaden the applicability of overlapping group sparsity in high‑dimensional learning tasks.

In summary, this work provides a theoretically sound, practically efficient solution to the overlapping group Lasso problem. By converting the proximal step into a smooth convex dual problem, it unlocks the power of fast gradient‑based optimizers and makes large‑scale, overlapping‑group regularization feasible for real‑world data analysis. The combination of rigorous analysis, algorithmic innovation, and thorough empirical evaluation makes the paper a valuable reference for researchers and practitioners dealing with structured sparsity in high‑dimensional settings.