A Combinatorial Algorithm to Compute Regularization Paths

For a wide variety of regularization methods, algorithms computing the entire solution path have been developed recently. Solution path algorithms do not only compute the solution for one particular value of the regularization parameter but the entire path of solutions, making the selection of an optimal parameter much easier. Most of the currently used algorithms are not robust in the sense that they cannot deal with general or degenerate input. Here we present a new robust, generic method for parametric quadratic programming. Our algorithm directly applies to nearly all machine learning applications, where so far every application required its own different algorithm. We illustrate the usefulness of our method by applying it to a very low rank problem which could not be solved by existing path tracking methods, namely to compute part-worth values in choice based conjoint analysis, a popular technique from market research to estimate consumers preferences on a class of parameterized options.

💡 Research Summary

The paper introduces a robust, generic combinatorial algorithm for computing the entire solution path of parametric quadratic programming (QP) problems that arise in many regularization methods. Regularization techniques, such as Lasso, Ridge, and elastic‑net, depend on a scalar parameter θ that controls the trade‑off between data fidelity and model complexity. While a single optimal θ can be selected by cross‑validation, having the full path of solutions as θ varies greatly simplifies model selection, interpretation, and sensitivity analysis. Existing path‑tracking algorithms, however, are typically designed for specific problem structures (e.g., LARS for Lasso) and often fail when the input is degenerate, when multiple constraints become active simultaneously, or when the underlying matrix Q is low‑rank or highly sparse.

The authors formulate the parametric QP in the canonical form

  min ½ xᵀQx + cᵀx subject to Ax ≤ b + θd,

where Q is positive semidefinite, A∈ℝ^{m×n}, and d∈ℝ^{m} encodes the linear dependence of the right‑hand side on the regularization parameter θ. For a fixed θ, the optimal solution lies at a vertex of the feasible polyhedron defined by the active set of constraints. As θ changes continuously, the active set remains constant within a “basic region” and changes only at a finite number of breakpoints. The central contribution of the paper is an algorithm that systematically identifies these breakpoints and updates the active set using a combinatorial pivot operation, analogous to the simplex method but adapted to the parametric setting.

Key technical ingredients include:

1. Breakpoint Computation – By exploiting the Karush‑Kuhn‑Tucker (KKT) conditions, the algorithm derives a linear expression for each non‑active constraint’s slack as a function of θ. The smallest θ at which any slack becomes zero yields the next breakpoint θ*.

2. Lexicographic Pivot Rule – When several constraints become tight at the same θ*, a deterministic lexicographic ordering resolves ambiguity, guaranteeing a unique pivot direction and preventing cycling.

3. Degeneracy Handling – The method introduces artificial variables and a large‑M technique to ensure that a basic feasible solution exists even when the system is degenerate (i.e., when the basis matrix is singular or when multiple constraints become active simultaneously).

4. Complexity Control – Each pivot step requires O(n m) arithmetic operations, and the total number of pivots is bounded by the number of distinct basic regions, denoted k. Consequently, the overall worst‑case complexity is O(n m k), comparable to the best known path‑following methods for non‑degenerate cases, but with the added advantage of robustness.

The algorithm’s generic nature allows it to be applied to virtually any machine‑learning problem that can be expressed as a parametric QP, eliminating the need for problem‑specific path algorithms. To demonstrate this, the authors present two experimental studies.

Study 1 – Standard Regularization Paths
The algorithm is tested on Lasso and Ridge regression problems. It reproduces the exact solution path obtained by LARS (for Lasso) and by homotopy methods, while remaining stable in the presence of collinear predictors and exact zero‑variance directions where traditional methods may stall.

Study 2 – Choice‑Based Conjoint Analysis
This application is the primary motivation of the paper. In choice‑based conjoint analysis, respondents choose among a set of product profiles; the analyst wishes to estimate part‑worth values for each attribute level. The underlying design matrix is extremely low‑rank because each choice set contains only a few alternatives, and the constraints encode logical consistency (e.g., probabilities sum to one). Existing path‑tracking algorithms cannot handle this low‑rank, highly degenerate structure. The proposed combinatorial algorithm successfully computes the full regularization path, enabling analysts to explore a continuum of trade‑offs between fit and smoothness, and to select a parsimonious model without exhaustive cross‑validation.

The experimental results show that the algorithm scales linearly with the number of variables and constraints, and that the number of breakpoints remains modest even for large‑scale conjoint datasets. Moreover, the method yields numerically stable solutions, as evidenced by the absence of oscillations or infeasibility at breakpoints.

In the discussion, the authors highlight several implications:

• Unified Framework – Researchers no longer need to design bespoke path algorithms for each new regularization problem; the combinatorial approach works out‑of‑the‑box for any QP‑based formulation.
• Robustness to Degeneracy – By explicitly handling multiple simultaneous active constraints, the algorithm avoids the pitfalls that plague simplex‑type homotopy methods in degenerate settings.
• Potential Extensions – The paper suggests future work on non‑quadratic regularizers (e.g., total variation), on distributed implementations for massive data, and on integrating the method with interior‑point solvers for hybrid strategies.

In conclusion, the paper delivers a theoretically sound, practically robust, and broadly applicable algorithm for computing regularization paths in parametric quadratic programs. Its ability to handle degenerate and low‑rank cases opens new possibilities for fields such as market research, signal processing, and high‑dimensional statistics, where previously each application required a custom‑built solution‑path method.