DC approximation approaches for sparse optimization

Sparse optimization refers to an optimization problem involving the zero-norm in objective or constraints. In this paper, nonconvex approximation approaches for sparse optimization have been studied with a unifying point of view in DC (Difference of Convex functions) programming framework. Considering a common DC approximation of the zero-norm including all standard sparse inducing penalty functions, we studied the consistency between global minimums (resp. local minimums) of approximate and original problems. We showed that, in several cases, some global minimizers (resp. local minimizers) of the approximate problem are also those of the original problem. Using exact penalty techniques in DC programming, we proved stronger results for some particular approximations, namely, the approximate problem, with suitable parameters, is equivalent to the original problem. The efficiency of several sparse inducing penalty functions have been fully analyzed. Four DCA (DC Algorithm) schemes were developed that cover all standard algorithms in nonconvex sparse approximation approaches as special versions. They can be viewed as, an $\ell _{1}$-perturbed algorithm / reweighted-$\ell _{1}$ algorithm / reweighted-$\ell _{1}$ algorithm. We offer a unifying nonconvex approximation approach, with solid theoretical tools as well as efficient algorithms based on DC programming and DCA, to tackle the zero-norm and sparse optimization. As an application, we implemented our methods for the feature selection in SVM (Support Vector Machine) problem and performed empirical comparative numerical experiments on the proposed algorithms with various approximation functions.

💡 Research Summary

This paper tackles the notoriously difficult ℓ₀‑norm sparse optimization problem by embedding it into the Difference‑of‑Convex (DC) programming framework and by using DC approximations of the ℓ₀ term. The authors first introduce a unified DC approximation function ϕθ(x) that, depending on a parameter θ, can represent all the popular non‑convex sparsity‑inducing penalties such as ℓp (0<p<1), SCAD, Capped‑ℓ₁, logarithmic and exponential concave functions. By treating the original problem
  min { f(x,y)+λ‖x‖₀ : (x,y)∈K }
with a DC objective f and a polyhedral convex feasible set K, they replace the ℓ₀ term with λϕθ(x) and study the relationship between the approximate problem and the original one.

The theoretical contributions are threefold. First, they prove a global‑optimality consistency result: for θ sufficiently large, any global minimizer of the approximate problem lies either exactly on a global minimizer of the original problem or within an ε‑neighbourhood of it. Second, they establish a local‑optimality link: when f is a DC function and K is polyhedral, some local minima of the approximate problem are also local minima of the original formulation. Third, in the special case where f is convex and bounded below on K, a subset of the approximate problem’s global solutions coincides exactly with the original problem’s global solutions.

A particularly strong result is obtained for the Capped‑ℓ₁ and SCAD penalties. By employing exact penalty techniques from DC programming, the authors show that there exists a threshold θ₀ such that for any θ>θ₀ the approximate problem is mathematically equivalent to the original ℓ₀ problem. When K is a box constraint, they even derive an explicit expression for θ₀, allowing practitioners to set the parameter without trial‑and‑error.

On the algorithmic side, the paper develops four DC Algorithm (DCA) schemes. The first scheme adds an ℓ₁ perturbation to the objective, thereby encompassing classical LASSO‑based re‑weighted ℓ₁ methods. The second and third schemes correspond to two variants of re‑weighted ℓ₁: one with fixed weights and one with weights updated at each iteration, both derived from a DC decomposition of the penalty. The fourth scheme handles the non‑concave piecewise‑linear DC approximation introduced in earlier work, solving it directly via DCA. All four schemes fit into the generic DCA framework, guaranteeing monotone descent of the objective and convergence to a critical point under standard assumptions. Each DCA iteration requires solving a convex subproblem, which can be efficiently handled by off‑the‑shelf solvers or proximal operators.

To validate the theory and the algorithms, the authors apply the four DCA variants to feature selection in Support Vector Machines (SVM). They test several sparsity‑inducing penalties (ℓ₁, Capped‑ℓ₁, SCAD, etc.) on high‑dimensional benchmark datasets (text classification, gene expression, etc.). The experiments measure classification accuracy, number of selected features, and computational time. Results demonstrate that Capped‑ℓ₁ and SCAD, when coupled with the proposed DCA, achieve higher sparsity and comparable or better predictive performance than traditional ℓ₁‑based methods. Moreover, the DCA‑based algorithms converge faster and exhibit lower sensitivity to the choice of θ, confirming the practical advantage of the exact‑penalty equivalence results.

In summary, the paper makes a substantial contribution by (i) providing rigorous global and local consistency guarantees for a broad class of non‑convex ℓ₀ approximations, (ii) identifying precise parameter regimes where the approximate and original problems are exactly equivalent, (iii) unifying a wide range of existing sparse‑optimization algorithms under a single DC programming and DCA umbrella, and (iv) delivering empirical evidence of superior performance on a realistic machine‑learning task. The work paves the way for further extensions to more complex constraints (e.g., structured sparsity, rank constraints) and for the development of scalable DC‑based solvers in large‑scale data analytics.