When is there a representer theorem? Vector versus matrix regularizers

We consider a general class of regularization methods which learn a vector of parameters on the basis of linear measurements. It is well known that if the regularizer is a nondecreasing function of the inner product then the learned vector is a linear combination of the input data. This result, known as the {\em representer theorem}, is at the basis of kernel-based methods in machine learning. In this paper, we prove the necessity of the above condition, thereby completing the characterization of kernel methods based on regularization. We further extend our analysis to regularization methods which learn a matrix, a problem which is motivated by the application to multi-task learning. In this context, we study a more general representer theorem, which holds for a larger class of regularizers. We provide a necessary and sufficient condition for these class of matrix regularizers and highlight them with some concrete examples of practical importance. Our analysis uses basic principles from matrix theory, especially the useful notion of matrix nondecreasing function.

💡 Research Summary

The paper investigates the precise conditions under which a representer theorem holds for regularized learning problems, first for vector‑valued parameters and then for matrix‑valued parameters that arise in multi‑task learning. In the classical setting, one seeks a vector (w\in\mathbb{R}^d) that minimizes a loss depending on linear measurements (\langle w,x_i\rangle) together with a regularizer (\Omega(w)). It has long been known that if the regularizer can be written as a non‑decreasing function of the squared norm, i.e. (\Omega(w)=g(\langle w,w\rangle)) with (g) monotone non‑decreasing, then any optimal solution lies in the span of the training inputs – the representer theorem. What was missing, however, was a proof that this condition is also necessary.

Using sub‑gradient optimality conditions and Lagrange multiplier analysis, the authors show that if (\Omega) does not depend solely on the inner product or if (g) contains a decreasing region, one can construct examples where the optimal solution has components orthogonal to the data span. Hence the “inner‑product‑only + monotone” form is both necessary and sufficient for the vector case.

The second part of the work extends the analysis to matrix‑valued parameters (W\in\mathbb{R}^{d\times T}), which model (T) related tasks simultaneously. The loss now depends on the set of linear measurements (\langle w_t,x_i\rangle) for each task (t), and the regularizer is a function of the whole matrix. The authors propose a natural generalisation: (\Omega(W)=h(W^{\top}W)), where (h) maps the symmetric positive‑semidefinite matrix (W^{\top}W) to a scalar. They introduce the concept of a matrix non‑decreasing function: for any two PSD matrices (A\preceq B) (i.e., (B-A) is PSD), the inequality (h(A)\le h(B)) holds.

The main theorem states that the matrix‑valued representer theorem holds if and only if the regularizer can be expressed in the above form with (h) matrix‑non‑decreasing. Under this condition, any optimal (W^\star) can be written as (W^\star = X A), where (X) contains the training inputs as columns and (A) is a coefficient matrix of size (n\times T). The proof relies on spectral function theory: a spectral function that is monotone with respect to the Loewner order preserves the order of eigenvalues, which translates into the required orthogonality properties of the optimality conditions. Conversely, if (h) fails to be monotone in the Loewner sense, the authors construct counter‑examples where the optimal solution leaves the column space of (X).

To illustrate the theory, the paper lists concrete regularizers that satisfy the conditions. For vectors, classic (\ell_2) regularization, higher‑order (\ell_p) norms with (p\ge2), and smooth functions such as (\lambda\log(1+|w|2^2)) all have monotone (g). For matrices, the trace norm (nuclear norm), the Frobenius norm, and group‑lasso‑type penalties (\lambda\sum{j=1}^d|W_{j,:}|_2) are shown to be matrix‑non‑decreasing, because they are spectral functions that increase with the eigenvalues of (W^{\top}W).

The practical impact of these results is twofold. First, when the representer theorem holds, the original high‑dimensional optimization can be reduced to a problem involving only the number of training samples (or tasks), leading to substantial computational savings and enabling the use of kernel tricks for nonlinear models. Second, in multi‑task settings, designers of regularizers now have a clear mathematical guideline: choose a regularizer that is a matrix‑non‑decreasing function of (W^{\top}W) to guarantee that the solution respects the shared structure encoded in the data. This bridges a gap between abstract functional analysis and concrete algorithm design, providing a solid foundation for future work on structured regularization, collaborative filtering, and deep models with shared parameters.

In summary, the authors complete the characterization of when representer theorems hold: for vectors, the regularizer must be a non‑decreasing function of the inner product; for matrices, it must be a matrix‑non‑decreasing function of the Gram matrix (W^{\top}W). The paper’s blend of elementary matrix theory, convex analysis, and illustrative examples makes the results both rigorous and readily applicable to modern machine‑learning practice.