An Exponential Lower Bound on the Complexity of Regularization Paths

For a variety of regularized optimization problems in machine learning, algorithms computing the entire solution path have been developed recently. Most of these methods are quadratic programs that are parameterized by a single parameter, as for example the Support Vector Machine (SVM). Solution path algorithms do not only compute the solution for one particular value of the regularization parameter but the entire path of solutions, making the selection of an optimal parameter much easier. It has been assumed that these piecewise linear solution paths have only linear complexity, i.e. linearly many bends. We prove that for the support vector machine this complexity can be exponential in the number of training points in the worst case. More strongly, we construct a single instance of n input points in d dimensions for an SVM such that at least \Theta(2^{n/2}) = \Theta(2^d) many distinct subsets of support vectors occur as the regularization parameter changes.

💡 Research Summary

The paper addresses a fundamental question about the computational complexity of solution‑path algorithms for regularized learning problems, focusing on the binary Support Vector Machine (SVM) with a single regularization parameter C. Historically, researchers have observed that the optimal SVM hyperplane changes piecewise‑linearly as C varies, and it has been widely assumed that the number of linear pieces (or “bends”) grows at most linearly with the number of training examples. This assumption underlies many path‑tracking methods that aim to compute the entire regularization path in a single run, thereby simplifying model selection.

The authors overturn this assumption by proving a worst‑case exponential lower bound on the number of distinct support‑vector sets that can appear along the path. Their main theorem states that there exists a single SVM instance with n training points embedded in d = n/2 dimensions such that the regularization path contains Θ(2^{n/2}) = Θ(2^{d}) different linear segments. Consequently, the path complexity can be exponential in the input size, not merely linear.

To establish the result, the paper first reformulates the SVM dual problem as a parametric linear program whose feasible region is a convex polytope. As C varies from 0 to ∞, the optimal solution traverses vertices of this polytope; each vertex corresponds to a particular set of support vectors, and a change of vertex occurs exactly when C crosses a critical value. Therefore, the number of vertices visited equals the number of bends in the solution path.

The core construction places the n points in a highly symmetric configuration in d‑dimensional space. The points are split into two opposite clusters, each containing d points with opposite class labels. By carefully choosing the inter‑cluster distances and the orientation of each point, the authors ensure that for every subset S ⊆ {1,…,d} there exists a range of C for which the support‑vector set is precisely the union of the points indexed by S together with the opposite‑cluster points that are forced to be support vectors by the geometry. In other words, each of the 2^{d} possible subsets of the d “free” points becomes the active support‑vector set for some interval of C. The construction respects the general‑position requirement (no degenerate alignments) by adding infinitesimal perturbations, guaranteeing that each transition is unique and that the path does not skip any of the 2^{d} vertices.

The proof proceeds by (1) showing that the constructed polytope indeed has 2^{d} vertices, (2) demonstrating that the objective function’s slope with respect to C is monotone across adjacent vertices, and (3) establishing that the critical values of C are distinct and ordered, which yields a strictly increasing sequence of intervals each associated with a different support‑vector set. This yields the exponential lower bound.

Empirical validation is provided on low‑dimensional instances (d ≤ 6) where the full exponential number of bends can be enumerated, confirming the theoretical prediction. Randomly generated data, by contrast, typically exhibit only a linear or near‑linear number of bends, illustrating that the exponential behavior is pathological but possible.

The paper’s implications are threefold. First, it cautions against assuming that path‑following algorithms will always run in polynomial time; in the worst case they may require exponential time and memory. Second, it highlights a limitation of single‑parameter regularization for model selection: computing the exact entire path may be infeasible for high‑dimensional data. Third, it opens new research directions, such as extending the analysis to multi‑parameter regularization (e.g., elastic‑net) or designing approximation schemes that avoid enumerating all bends while still providing useful information for hyperparameter tuning.

In summary, the authors provide a rigorous construction that demonstrates an exponential lower bound on the complexity of SVM solution paths, thereby refuting the long‑standing belief in linear path complexity and reshaping our understanding of the theoretical limits of regularization‑path algorithms.