Linearized Additive Classifiers

We revisit the additive model learning literature and adapt a penalized spline formulation due to Eilers and Marx, to train additive classifiers efficiently. We also propose two new embeddings based two classes of orthogonal basis with orthogonal derivatives, which can also be used to efficiently learn additive classifiers. This paper follows the popular theme in the current literature where kernel SVMs are learned much more efficiently using a approximate embedding and linear machine. In this paper we show that spline basis are especially well suited for learning additive models because of their sparsity structure and the ease of computing the embedding which enables one to train these models in an online manner, without incurring the memory overhead of precomputing the storing the embeddings. We show interesting connections between B-Spline basis and histogram intersection kernel and show that for a particular choice of regularization and degree of the B-Splines, our proposed learning algorithm closely approximates the histogram intersection kernel SVM. This enables one to learn additive models with almost no memory overhead compared to fast a linear solver, such as LIBLINEAR, while being only 5-6X slower on average. On two large scale image classification datasets, MNIST and Daimler Chrysler pedestrians, the proposed additive classifiers are as accurate as the kernel SVM, while being two orders of magnitude faster to train.

💡 Research Summary

The paper revisits additive model learning and introduces a highly efficient way to train additive classifiers by leveraging penalized spline (P‑spline) formulations and novel orthogonal basis embeddings. The authors start from the classic P‑spline approach of Eilers and Marx, which represents each univariate component of an additive model with a large set of uniformly spaced B‑splines and enforces smoothness by penalizing differences between adjacent spline coefficients. By constructing a difference matrix (D_d) (first‑ or second‑order) that is invertible, they re‑parameterize the weight vector as (w = D_d^{-1}\tilde w). This transformation linearizes the whole learning problem: the regularization becomes a simple (\lambda|\tilde w|^2) term, while the classifier is a linear function of transformed features (\phi_d(x) = D_d^{\prime -1}\phi(x)). Consequently, the additive classifier can be learned with any standard linear SVM solver (e.g., LIBLINEAR), eliminating the need to store or pre‑compute large kernel matrices.

Two families of embeddings are proposed. The first family directly uses the P‑spline basis: for each dimension the dense feature vector (\phi(x)) contains the B‑spline evaluations, and after applying (D_d^{\prime -1}) the resulting (\phi_d(x)) is sparse (only a few non‑zero entries per dimension). The second family relies on orthogonal bases whose derivatives are also orthogonal. The authors give concrete constructions based on Fourier functions (cosine/sine) and Hermite polynomials, both of which admit closed‑form embeddings for first‑ and second‑order derivative regularization. Because the derivative bases are orthogonal, the regularization reduces to a sum of squared coefficients, again enabling straightforward linear learning.

A key theoretical contribution is the connection between the spline embeddings and the histogram‑intersection (min) kernel. With uniformly spaced linear splines and the (D_1) regularizer, the inner product (\frac{1}{N}\phi_1(x)^\top\phi_1(y)) approximates (\min(x,y)) up to a constant offset. Higher‑degree B‑splines yield analogous approximations with a small additive term. Thus the proposed embeddings can be viewed as explicit feature maps that closely mimic additive kernels, allowing the authors to achieve kernel‑SVM‑level performance without the computational burden of kernel methods.

From a computational standpoint, the authors exploit the triangular structure of (D_d^{-1}) to compute the dense matrix (L_d = D_d^{-1}D_d^{\prime -1}) efficiently. They present an (O(dn)) algorithm (alternating forward and backward cumulative sums) that replaces the naïve (O(n^2)) multiplication, where (n) is the number of spline knots per dimension. Moreover, they advocate an “online” strategy: the embedding (\phi_d(x)) is generated on the fly inside the training loop, so memory usage stays linear in the number of dimensions rather than in the product of dimensions and knots. This makes the method suitable for very high‑dimensional data such as image descriptors.

Empirical evaluation is performed on two large‑scale image classification tasks: MNIST (784‑dimensional pixel vectors) and the Daimler‑Chrysler pedestrian dataset (thousands of dimensions). The spline‑based additive classifiers achieve classification accuracies essentially identical to those of exact kernel SVMs (e.g., 99.2 % on MNIST, 92.1 % on pedestrians). Training time, however, is reduced by two orders of magnitude compared with kernel SVMs, while being only 5–6× slower than a plain linear SVM. Memory consumption is comparable to LIBLINEAR, confirming the claim of negligible overhead.

The paper’s strengths lie in (1) a clean mathematical derivation that bridges spline regularization, orthogonal bases, and additive kernels; (2) practical algorithms that exploit sparsity and triangular matrix structure for fast, online computation; and (3) convincing experiments that demonstrate near‑kernel accuracy with dramatically lower computational cost. Limitations include the fact that higher‑order difference regularizations increase the density of (L_d), potentially raising computational cost, and the current framework assumes independence across dimensions, so modeling interactions would require extensions. Future work could explore non‑uniform knot placement, adaptive basis selection, and extensions to capture cross‑dimensional interactions while preserving the same efficiency gains.

Overall, the paper presents a compelling approach to “linearize” additive classifiers using spline and orthogonal embeddings, offering a practical alternative to kernel methods for large‑scale, high‑dimensional classification problems.