Structured sparsity through convex optimization

Sparse estimation methods are aimed at using or obtaining parsimonious representations of data or models. While naturally cast as a combinatorial optimization problem, variable or feature selection admits a convex relaxation through the regularization by the $\ell_1$-norm. In this paper, we consider situations where we are not only interested in sparsity, but where some structural prior knowledge is available as well. We show that the $\ell_1$-norm can then be extended to structured norms built on either disjoint or overlapping groups of variables, leading to a flexible framework that can deal with various structures. We present applications to unsupervised learning, for structured sparse principal component analysis and hierarchical dictionary learning, and to supervised learning in the context of non-linear variable selection.

💡 Research Summary

The paper “Structured Sparsity through Convex Optimization” addresses a fundamental limitation of the classic ℓ₁‑norm regularization: while it promotes sparsity, it treats each variable independently and ignores any known relationships among variables. In many scientific and engineering applications, prior knowledge about spatial proximity, hierarchical organization, or functional grouping is available and can be leveraged to improve both interpretability and predictive performance. The authors propose a unified convex‑optimization framework that extends the ℓ₁ penalty to structured sparsity‑inducing norms.

The core construction is a mixed ℓ₁/ℓ_q norm defined on groups of variables. Let G be a collection of groups (a partition of the index set for disjoint groups, or an overlapping collection for more complex structures). For each group g a positive weight d_g is assigned, and the regularizer is
Ω(w) = Σ_{g∈G} d_g ‖w_g‖_q, q∈(1,∞].
When q=2 the norm reduces to the well‑known group Lasso, and q=∞ yields a max‑norm within each group. This formulation forces all variables in a group to be selected or discarded together, thereby encoding the desired structural prior.

Two major scenarios are considered:

1. Disjoint groups – The groups form a partition of the variables. The mixed norm directly yields block‑wise sparsity and has been shown to improve prediction and interpretability when the block structure reflects the underlying problem (e.g., spatially contiguous voxels in neuro‑imaging).

2. Overlapping groups – Real‑world problems often involve variables belonging to several groups simultaneously (e.g., genes participating in multiple pathways, pixels belonging to several overlapping patches). The authors present two complementary constructions to handle overlaps while preserving convexity:
   a. Variable duplication – Each variable is duplicated for each group it belongs to; the mixed norm is applied to the duplicated vector, and an additional linear constraint enforces consistency among the copies.
   b. Latent variable formulation – The original variable vector w is expressed as a linear transformation of a latent vector v; the structured norm is imposed on v, and the transformation guarantees that the resulting w respects the overlapping group structure.

Both constructions lead to a proximal operator that can be computed efficiently (group‑wise ℓ₂ or ℓ_∞ projections), enabling the use of fast first‑order methods such as FISTA or ADMM. The paper provides convergence guarantees for these algorithms and derives statistical results: under suitable restricted‑isometry‑type conditions, the estimators enjoy variable‑selection consistency and bounded prediction error, even in the presence of overlapping groups.

The authors illustrate the versatility of the framework through three major applications:

• Structured Sparse Principal Component Analysis (PCA) – By adding the mixed norm to the PCA objective, the resulting principal components are forced to respect a pre‑specified spatial or hierarchical pattern, yielding more interpretable components for image and brain‑signal data.

• Hierarchical Dictionary Learning – Dictionaries are organized as trees; the mixed norm with overlapping groups enforces that if a child atom is selected, its ancestors must also be selected, producing a natural hierarchical representation of signals.

• Non‑linear Variable Selection – In kernel‑based learning, each kernel can be treated as a group. Overlapping group regularization selects a sparse set of kernels (or features) while respecting prior relationships, which is particularly useful in genomics where SNPs belong to multiple functional pathways.

Empirical evaluations on fMRI, face‑recognition, and array‑CGH genomic data demonstrate that structured sparsity consistently outperforms plain ℓ₁ regularization in terms of prediction accuracy, sparsity pattern stability, and interpretability. The computational overhead introduced by duplication or latent‑variable formulations is modest thanks to efficient proximal implementations.

The paper’s contributions are threefold: (i) a principled convex formulation of a broad class of structured sparsity penalties; (ii) scalable algorithms with provable convergence and statistical guarantees; (iii) concrete demonstrations that embedding domain knowledge via these penalties yields tangible benefits across diverse supervised and unsupervised learning tasks. Limitations include the need for a priori specification of groups and weights, and the potential increase in computational cost for highly overlapping, large‑scale problems. Future work may explore automatic group discovery, adaptive weight learning, and distributed optimization to further broaden the applicability of structured sparsity.