A General Theory of Concave Regularization for High Dimensional Sparse Estimation Problems

Concave regularization methods provide natural procedures for sparse recovery. However, they are difficult to analyze in the high dimensional setting. Only recently a few sparse recovery results have been established for some specific local solutions obtained via specialized numerical procedures. Still, the fundamental relationship between these solutions such as whether they are identical or their relationship to the global minimizer of the underlying nonconvex formulation is unknown. The current paper fills this conceptual gap by presenting a general theoretical framework showing that under appropriate conditions, the global solution of nonconvex regularization leads to desirable recovery performance; moreover, under suitable conditions, the global solution corresponds to the unique sparse local solution, which can be obtained via different numerical procedures. Under this unified framework, we present an overview of existing results and discuss their connections. The unified view of this work leads to a more satisfactory treatment of concave high dimensional sparse estimation procedures, and serves as guideline for developing further numerical procedures for concave regularization.

💡 Research Summary

The paper addresses a long‑standing gap in the theory of concave (non‑convex) regularization for high‑dimensional sparse estimation. While concave penalties such as SCAD, MCP, and log‑penalty are known to reduce bias compared with the convex (\ell_{1}) (Lasso) penalty, their non‑convex nature makes it difficult to guarantee that a computed solution is globally optimal or even that a global optimum has desirable statistical properties. The authors develop a unified theoretical framework that simultaneously establishes (i) statistical recovery guarantees for the global minimizer of a broad class of concave regularized loss functions, and (ii) conditions under which this global minimizer coincides with a unique sparse local solution that can be obtained by a variety of practical algorithms.

The analysis begins with a standard linear model (y = X\beta^{} + \varepsilon) where the true coefficient vector (\beta^{}) is (s)-sparse. The loss function is the usual squared error, and the penalty is a generic concave function (\rho_{\lambda}(|\beta|)) satisfying mild regularity conditions: (a) (\rho_{\lambda}(0)=0), (b) (\rho’{\lambda}(t)) is decreasing and bounded by (\lambda), and (c) (\rho{\lambda}) is Lipschitz‑continuous on (