Optimization with Sparsity-Inducing Penalties

Sparse estimation methods are aimed at using or obtaining parsimonious representations of data or models. They were first dedicated to linear variable selection but numerous extensions have now emerged such as structured sparsity or kernel selection. It turns out that many of the related estimation problems can be cast as convex optimization problems by regularizing the empirical risk with appropriate non-smooth norms. The goal of this paper is to present from a general perspective optimization tools and techniques dedicated to such sparsity-inducing penalties. We cover proximal methods, block-coordinate descent, reweighted $\ell_2$-penalized techniques, working-set and homotopy methods, as well as non-convex formulations and extensions, and provide an extensive set of experiments to compare various algorithms from a computational point of view.

💡 Research Summary

This paper provides a comprehensive survey of optimization techniques tailored for sparsity‑inducing penalties, covering a wide range of regularizers such as the ℓ₁ norm, mixed ℓ₁/ℓ_q norms, structured sparsity norms, and multiple kernel learning (MKL) formulations. It begins by establishing the mathematical foundations: definitions of the various norms, their geometric interpretation via unit balls, and the necessary convex analysis tools—including subgradients, Fenchel duality, and quadratic variational representations—that enable the derivation of optimality conditions for non‑smooth objectives.

The authors then systematically categorize algorithmic approaches. Proximal methods, including ISTA and its accelerated variant FISTA, are presented as the cornerstone for handling non‑smooth regularizers; detailed proximal operators for each norm are derived, and extensions to structured MKL are discussed. Block‑coordinate descent (BCD) exploits separability across groups or blocks, offering efficient updates and strong convergence guarantees, especially for ℓ₁/ℓ₂ mixed norms. Weighted‑ℓ₂ (re‑weighted least squares) algorithms are introduced via the quadratic variational form, converting ℓ₁‑type penalties into a sequence of weighted ℓ₂ problems that are particularly efficient for squared‑loss settings.

Working‑set and homotopy techniques leverage the sparsity of the solution to progressively enlarge the active set or trace the entire regularization path as the penalty parameter λ varies. The classic LARS algorithm for the Lasso exemplifies homotopy, while generic working‑set meta‑algorithms can be combined with any of the previously described solvers. The paper also surveys non‑convex strategies: greedy methods such as Orthogonal Matching Pursuit and forward selection, DC‑programming based re‑weighted ℓ₁ schemes, sparse matrix factorization and dictionary learning approaches, and Bayesian formulations that interpret sparsity through hierarchical priors.

A substantial experimental section benchmarks all these methods on synthetic and real datasets for three scenarios: plain Lasso, group sparsity, and more complex structured sparsity. Metrics include convergence speed, computational cost, and memory usage. Results show that proximal gradient and block‑coordinate descent dominate in most settings, working‑set/homotopy excel when a full regularization path is required, and re‑weighted ℓ₂ methods are highly competitive for least‑squares losses. The authors conclude by summarizing the trade‑offs, highlighting open challenges such as theoretical analysis of non‑convex penalties, large‑scale distributed implementations, and integration with deep learning architectures. Overall, the paper serves as a detailed guide for researchers and practitioners to select and implement the most appropriate optimization algorithm for a given sparsity‑inducing regularizer.