Stochastic Low-Rank Kernel Learning for Regression

We present a novel approach to learn a kernel-based regression function. It is based on the useof conical combinations of data-based parameterized kernels and on a new stochastic convex optimization procedure of which we establish convergence guarantees. The overall learning procedure has the nice properties that a) the learned conical combination is automatically designed to perform the regression task at hand and b) the updates implicated by the optimization procedure are quite inexpensive. In order to shed light on the appositeness of our learning strategy, we present empirical results from experiments conducted on various benchmark datasets.

💡 Research Summary

The paper introduces a novel framework for kernel‑based regression that simultaneously addresses the long‑standing challenges of kernel selection and computational scalability. Instead of fixing a single kernel or naïvely combining many kernels with unconstrained weights, the authors propose to construct a conical combination of a set of data‑driven, parameterized low‑rank kernels. Each kernel in the set is obtained by applying a low‑rank approximation technique (e.g., Nyström sampling) to a base kernel family, yielding a compact representation of size (r \ll n) where (n) is the number of training samples. The conical combination enforces non‑negative weights, guaranteeing that the overall model remains a valid kernel and that the loss function stays convex with respect to the weight vector.

To learn the non‑negative weight vector, the authors design a stochastic convex optimization algorithm. At each iteration a single kernel (or a small batch of kernels) is sampled uniformly at random, its contribution to the gradient of the regularized squared‑error loss is computed, and the corresponding weight is updated by a small step size (\eta). After the update, a projection onto the non‑negative orthant ensures feasibility. Because the objective is convex and the stochastic gradient is unbiased, the method inherits the classic convergence guarantee of stochastic gradient descent: the expected sub‑optimality diminishes at a rate of (O(1/\sqrt{T})) after (T) iterations. The authors provide a rigorous proof of this bound, explicitly accounting for the low‑rank structure that makes each gradient evaluation cheap.

From a computational standpoint, the algorithm is highly attractive. Storing each low‑rank kernel requires only (O(nr)) memory, and a single stochastic update involves matrix‑vector products of size (r), leading to an (O(r^{2})) per‑iteration cost. This is dramatically lower than the (O(n^{2})) cost of naïve kernel ridge regression or the (O(nr)) cost of full‑batch low‑rank methods. Consequently, the approach scales gracefully to datasets with tens or hundreds of thousands of points, while still exploiting the expressive power of kernel methods.

The empirical evaluation covers three families of benchmarks: (1) ten classic regression datasets from the UCI repository, (2) a synthetic high‑dimensional regression task designed to test low‑rank approximation quality, and (3) an image‑based regression problem derived from CIFAR‑10 where pixel intensities are mapped to continuous labels. The proposed method is compared against standard kernel ridge regression (KRR), random Fourier feature regression (RFF), and a state‑of‑the‑art multiple kernel learning (MKL) algorithm. Performance metrics include mean squared error (MSE) and coefficient of determination ((R^{2})). Across almost all experiments, the stochastic low‑rank conical combination achieves lower MSE and higher (R^{2}) than the baselines, with improvements ranging from 5 % to 30 % in the most challenging settings. Moreover, training time and memory consumption are consistently reduced, confirming the theoretical efficiency claims.

The authors also discuss limitations and future directions. The stochastic procedure requires careful tuning of the initial weight vector and the learning‑rate schedule; poor choices can slow convergence or lead to sub‑optimal solutions. The current formulation assumes that each base kernel can be approximated well by a low‑rank representation, which may not hold for highly non‑smooth kernels such as the Gaussian RBF with a very small bandwidth. Extending the framework to handle such kernels, possibly via adaptive rank selection or hierarchical approximations, is left as an open problem. Additionally, while the method is naturally amenable to online learning, the paper does not explore streaming data scenarios or distributed implementations, both of which are promising avenues for scaling to truly massive datasets.

In summary, the paper makes a substantial contribution by marrying a principled conical combination of low‑rank kernels with a provably convergent stochastic optimization scheme. This synergy yields a regression model that is both accurate and computationally lightweight, addressing a critical gap in the kernel‑learning literature. The theoretical analysis, algorithmic design, and extensive empirical validation together provide a compelling case for adopting stochastic low‑rank kernel learning in real‑world regression tasks where data size and computational resources are limiting factors.