Kernel Partial Least Squares is Universally Consistent

We prove the statistical consistency of kernel Partial Least Squares Regression applied to a bounded regression learning problem on a reproducing kernel Hilbert space. Partial Least Squares stands out of well-known classical approaches as e.g. Ridge Regression or Principal Components Regression, as it is not defined as the solution of a global cost minimization procedure over a fixed model nor is it a linear estimator. Instead, approximate solutions are constructed by projections onto a nested set of data-dependent subspaces. To prove consistency, we exploit the known fact that Partial Least Squares is equivalent to the conjugate gradient algorithm in combination with early stopping. The choice of the stopping rule (number of iterations) is a crucial point. We study two empirical stopping rules. The first one monitors the estimation error in each iteration step of Partial Least Squares, and the second one estimates the empirical complexity in terms of a condition number. Both stopping rules lead to universally consistent estimators provided the kernel is universal.

💡 Research Summary

This paper establishes the statistical consistency of kernel Partial Least Squares (PLS) regression for bounded regression problems defined on a reproducing kernel Hilbert space (RKHS). Unlike classical linear estimators such as ridge regression or principal components regression, kernel PLS does not arise from minimizing a global cost function over a fixed model. Instead, it builds approximate solutions by projecting the response onto a sequence of data‑dependent subspaces that are nested and grow with each iteration. The authors exploit a well‑known equivalence between PLS and the conjugate gradient (CG) algorithm applied to the normal equations of the kernel ridge system. This equivalence allows them to view each PLS iteration as a CG step in a Krylov subspace, and consequently to treat early stopping as a regularization mechanism.

The central theoretical contribution is a proof that, when the kernel is universal (i.e., its RKHS is dense in the space of continuous functions on the input domain), kernel PLS equipped with an appropriate stopping rule yields a universally consistent estimator. “Universal consistency” means that as the sample size (n) tends to infinity, the expected risk of the estimator converges to the Bayes risk, regardless of the underlying data‑generating distribution.

Two empirical stopping criteria are investigated. The first monitors the empirical estimation error at each iteration: the algorithm stops when the decrease in the empirical mean‑squared error falls below a pre‑specified tolerance. This rule directly tracks the trade‑off between bias reduction and variance inflation. The second criterion estimates the empirical complexity of the current Krylov subspace via its condition number. By computing the ratio of the largest to the smallest eigenvalue of the subspace’s Gram matrix, the algorithm stops when this condition number exceeds a threshold, thereby preventing numerical instability and over‑fitting. Both criteria are shown to select a number of iterations (m_n) that grows with (n) at a rate sufficient to guarantee consistency.

The proof proceeds in three main steps. First, the authors establish that the data‑dependent Krylov subspaces converge to the full RKHS as the sample size increases, using concentration inequalities for kernel Gram matrices. Second, they bound the generalization error of the CG‑based estimator by combining Rademacher complexity arguments with matrix concentration results, showing that the error decays at the order (O(n^{-1/2})) provided the stopping rule controls the subspace’s spectral properties. Third, they demonstrate that the two proposed stopping rules indeed enforce the required spectral conditions: the error‑monitoring rule stops when the residual norm becomes sufficiently small, while the condition‑number rule stops before the smallest eigenvalue drops below a level proportional to (1/n).

The analysis highlights several practical implications. Kernel PLS simultaneously performs dimensionality reduction and regression, making it computationally attractive for high‑dimensional, non‑linear problems. Early stopping eliminates the need for an explicit regularization parameter, simplifying model selection. The universality assumption on the kernel is essential; non‑universal kernels may lead to subspaces that fail to approximate the target function adequately, breaking consistency.

In conclusion, the paper provides a rigorous foundation for using kernel PLS as a statistically sound learning method. By linking PLS to CG and carefully designing data‑driven stopping rules, the authors show that kernel PLS attains the same universal consistency guarantees as more traditional kernel methods, while retaining its unique algorithmic advantages. Future work may explore empirical validation, extensions to multi‑output or classification settings, and integration with robust loss functions.