Learning with Structured Sparsity

This paper investigates a new learning formulation called structured sparsity, which is a natural extension of the standard sparsity concept in statistical learning and compressive sensing. By allowing arbitrary structures on the feature set, this concept generalizes the group sparsity idea that has become popular in recent years. A general theory is developed for learning with structured sparsity, based on the notion of coding complexity associated with the structure. It is shown that if the coding complexity of the target signal is small, then one can achieve improved performance by using coding complexity regularization methods, which generalize the standard sparse regularization. Moreover, a structured greedy algorithm is proposed to efficiently solve the structured sparsity problem. It is shown that the greedy algorithm approximately solves the coding complexity optimization problem under appropriate conditions. Experiments are included to demonstrate the advantage of structured sparsity over standard sparsity on some real applications.

💡 Research Summary

This paper presents a comprehensive framework for “structured sparsity,” a generalization of the standard sparsity concept in statistical learning and compressive sensing. Moving beyond the simple count of non-zero coefficients, structured sparsity incorporates prior knowledge about the relationships or patterns among features, such as group membership, hierarchical organization, or spatial connectivity.

The core theoretical contribution is the introduction of “coding complexity” as a unified measure to quantify structured sparsity. For a support set F (the set of indices with non-zero coefficients), a coding length cl(F) is defined, representing the number of bits needed to describe F under a chosen coding scheme that reflects the presumed structure. The overall structured sparse coding complexity for a coefficient vector β is then defined as c(β) = cl(supp(β)) + |supp(β)|. This formulation combines the cost of describing the pattern of non-zeros (the structure) with the cost of describing their values. The paper establishes that if the true underlying signal has low coding complexity, regularization methods promoting low coding complexity can achieve superior performance compared to standard sparsity-promoting methods like Lasso, requiring fewer measurements for accurate recovery.

To operationalize this theory for a wide range of structures, the paper introduces “block coding.” This scheme works by assigning coding lengths to a predefined set of base blocks (subsets of features). Any support set F can then be encoded as a union of these blocks. This framework elegantly encapsulates various specific structures: standard sparsity (using single-element blocks), (strong) group sparsity (using disjoint group blocks), tree-based sparsity (favoring hierarchical patterns), and a very general “graph sparsity.” Graph sparsity uses the connectivity of a graph defined over the features to efficiently encode connected components, which is highly relevant for applications like image processing where spatially contiguous regions are significant.

Addressing the computational challenge, the paper proposes a “structured greedy algorithm.” Instead of an intractable combinatorial search over all support sets, this algorithm greedily adds entire blocks from the block set B. When B contains a manageable number of base blocks, this algorithm is computationally efficient. The authors provide theoretical guarantees showing that under appropriate conditions, this greedy algorithm can approximately recover signals with low coding complexity.

In summary, this work makes several key advances: 1) It provides a general information-theoretic framework (coding complexity) to quantify and leverage structural assumptions in sparse learning. 2) It demonstrates theoretically that exploiting structure through coding complexity regularization can provably improve upon standard sparsity. 3) It proposes a flexible “block coding” scheme to model diverse structures. 4) It offers an efficient structured greedy algorithm to solve the resulting optimization problem. The paper thus bridges theory and practice, offering both a unifying perspective on prior work in group/tree sparsity and a pathway to model even more complex structural relationships in high-dimensional data.