Smoothing proximal gradient method for general structured sparse regression

We study the problem of estimating high-dimensional regression models regularized by a structured sparsity-inducing penalty that encodes prior structural information on either the input or output variables. We consider two widely adopted types of penalties of this kind as motivating examples: (1) the general overlapping-group-lasso penalty, generalized from the group-lasso penalty; and (2) the graph-guided-fused-lasso penalty, generalized from the fused-lasso penalty. For both types of penalties, due to their nonseparability and nonsmoothness, developing an efficient optimization method remains a challenging problem. In this paper we propose a general optimization approach, the smoothing proximal gradient (SPG) method, which can solve structured sparse regression problems with any smooth convex loss under a wide spectrum of structured sparsity-inducing penalties. Our approach combines a smoothing technique with an effective proximal gradient method. It achieves a convergence rate significantly faster than the standard first-order methods, subgradient methods, and is much more scalable than the most widely used interior-point methods. The efficiency and scalability of our method are demonstrated on both simulation experiments and real genetic data sets.

💡 Research Summary

The paper addresses the computational challenge of fitting high‑dimensional regression models that are regularized by structured sparsity‑inducing penalties. Two widely used penalties are taken as motivating examples: (1) the overlapping‑group‑lasso, which extends the classic group‑lasso by allowing variables to belong to multiple groups, and (2) the graph‑guided fused‑lasso, which extends the fused‑lasso by penalizing differences between coefficients of variables that are linked in a predefined graph. Both penalties encode valuable prior information about the relationships among predictors or responses, but their non‑separability and nonsmoothness make standard optimization techniques inefficient or inapplicable.

To overcome these difficulties, the authors propose the Smoothing Proximal Gradient (SPG) method. The core idea is to apply Nesterov’s smoothing technique to the nonsmooth regularizer. By introducing a smoothing parameter μ, the original regularizer R(β) is replaced with a smooth approximation R_μ(β) that is differentiable and has a Lipschitz‑continuous gradient. The original objective becomes

  L(β) = ℓ(β) + R_μ(β),

where ℓ(β) is any smooth convex loss (e.g., squared loss, logistic loss). Because R_μ is smooth, its gradient can be computed efficiently, while the original nonsmooth structure is retained through a proximal step that uses the exact proximal operator of R (not of R_μ). This hybrid approach yields a problem that is amenable to accelerated proximal gradient updates:

  β^{k+1} = prox_{α_k R}\bigl(β^k – α_k∇ℓ(β^k) – α_k∇R_μ(β^k)\bigr),

with a fixed step size α_k and Nesterov momentum for acceleration. Crucially, the proximal operator of the original regularizer remains tractable: for overlapping‑group‑lasso it reduces to group‑wise soft‑thresholding followed by averaging over overlapping variables; for graph‑guided fused‑lasso it can be solved by a total‑variation‑type algorithm (e.g., dynamic programming) in linear time.

Theoretical analysis shows that SPG attains a convergence rate of O(1/ε) in terms of the number of gradient evaluations needed to achieve an ε‑accurate solution. This is substantially faster than the O(1/ε²) rate of generic subgradient methods and avoids the cubic‑time complexity of interior‑point solvers, which become prohibitive for p in the tens of thousands. Moreover, the memory footprint of SPG is linear in the number of variables, making it suitable for large‑scale problems and for GPU acceleration.

Empirical evaluation consists of two parts. First, synthetic experiments vary the dimensionality from 1,000 to 100,000 variables, generate random overlapping groups and random graphs, and compare SPG against ADMM, FISTA, and plain subgradient descent. SPG consistently converges in fewer iterations and less wall‑clock time, and its performance is largely insensitive to the degree of overlap or graph density. Second, real‑world genetic data (e.g., SNP‑to‑gene association studies) are analyzed. Overlapping‑group‑lasso captures known functional gene sets, while graph‑guided fused‑lasso respects known interaction networks. On these datasets SPG achieves 5–10× speed‑ups over the best competing methods while delivering comparable or slightly better predictive accuracy (measured by R² for regression and AUC for classification).

In summary, the paper introduces a general, scalable optimization framework for structured sparse regression that combines smoothing of the nonsmooth penalty with an accelerated proximal gradient scheme. By preserving the exact proximal operator of the original regularizer, SPG leverages the structural information encoded in the penalty without sacrificing computational efficiency. The method works with any smooth convex loss, handles a broad class of penalties, and demonstrates both theoretical superiority and practical effectiveness. The authors suggest future extensions to non‑quadratic losses (logistic, Poisson), time‑varying graph structures, and multi‑task learning scenarios, indicating that the SPG paradigm could become a standard tool for a wide range of high‑dimensional structured estimation problems.