L0-Regularized Quadratic Surface Support Vector Machines

Kernel-free quadratic surface support vector machines (QSVM) have recently gained traction due to their flexibility in modeling nonlinear decision boundaries without relying on kernel functions. However, the introduction of a full quadratic classifier significantly increases the number of model parameters, scaling quadratically with data dimensionality, which often leads to overfitting and makes interpretation difficult. To address these challenges, we propose sparse variants of the QSVM by enforcing a cardinality constraint on the model parameters. While enhancing generalization and promoting sparsity, leveraging the $\ell_0$-norm inevitably incurs additional computational complexity. To tackle this, we develop a penalty decomposition algorithm capable of producing solutions that provably satisfy the first-order Lu-Zhang optimality conditions. We show that the subproblems arising within the algorithm either admit closed-form solutions or can be solved efficiently through dual formulations, which contributes to the method’s overall effectiveness. Besides, we analyze the convergence behavior of the algorithm under both loss settings. In addition, the numerical experiments on public benchmark datasets indicate that the proposed model is competitive with commonly used SVM variants and produces sparse solutions as expected. Moreover, its strong performance on real-world credit datasets demonstrates its potential for credit scoring applications.

💡 Research Summary

The paper addresses a fundamental limitation of kernel‑free quadratic surface support vector machines (QSVMs): the number of model parameters grows quadratically with the input dimension, leading to severe over‑fitting and poor interpretability. While previous work has reduced this burden by either imposing an ℓ₁ sparsity penalty or restricting the quadratic weight matrix to a diagonal form, both approaches either yield non‑unique solutions or discard valuable pairwise feature interactions. To overcome these drawbacks, the authors propose to directly enforce an ℓ₀‑norm cardinality constraint on the concatenated vector of the half‑vectorized quadratic matrix and the linear bias term. This constraint caps the number of non‑zero entries at a user‑specified level k, providing exact control over model complexity and enabling true feature selection.

Because ℓ₀ regularization renders the optimization problem combinatorial and intractable, the authors develop a Penalty Decomposition (PD) algorithm. They introduce an auxiliary variable u to decouple the ℓ₀ constraint from the smooth loss term, leading to a constrained formulation with a quadratic penalty ½ρ‖z − u‖². The algorithm alternates between (i) a z‑update that minimizes a smooth objective consisting of a quadratic form ½zᵀGz plus a loss term (either hinge loss H(t)=max(t,0) or squared loss H(t)=t²), and (ii) a u‑update that solves a pure ℓ₀‑constrained projection. The latter reduces to a hard‑thresholding operation: keep the k largest‑magnitude components of the current z and set the rest to zero. The z‑subproblem admits a closed‑form solution or can be solved efficiently via its dual because G is positive semidefinite. Convergence analysis shows that any limit point of the PD iterates satisfies the Lu‑Zhang first‑order optimality conditions, and that with a sufficiently large penalty parameter ρ the iterates approach a feasible solution of the original ℓ₀‑QSVM problem.

The authors present a unified model that covers both hinge‑loss QSVM and least‑squares QSVM under the same framework, and they detail the algorithmic steps, stopping criteria for inner and outer loops, and parameter update rules (e.g., geometric increase of ρ by factor β). Extensive experiments are conducted on public benchmark datasets (UCI binary classification tasks) and on a real‑world credit‑scoring dataset. The experiments systematically vary the sparsity budget k and the regularization constant C, reporting classification accuracy, F1‑score, AUC, and the number of selected features. Results demonstrate that the ℓ₀‑regularized QSVM achieves comparable or superior predictive performance to ℓ₁‑regularized QSVM and other standard SVM variants while using dramatically fewer features. In the credit‑scoring application, models with as few as 10–15 selected variables attain AUC values on par with full‑feature baselines, offering clear interpretability for domain experts. Computationally, the PD algorithm scales well: the inner subproblems are solved in closed form or via low‑dimensional duals, and the overall runtime grows roughly linearly with the sparsity budget.

In summary, the paper makes three key contributions: (1) it introduces exact ℓ₀ sparsity into kernel‑free QSVMs, enabling precise control of model complexity and genuine feature selection; (2) it devises a penalty‑decomposition algorithm with provable convergence to Lu‑Zhang stationary points, where each subproblem is either analytically tractable or efficiently solvable via duality; (3) it validates the approach empirically, showing competitive accuracy, strong sparsity, and practical applicability to credit risk modeling. The work opens avenues for further research, such as extensions to multi‑class problems, incorporation of more sophisticated loss functions, and adaptation to online or distributed learning settings.