Featured Reproducing Kernel Banach Spaces for Learning and Neural Networks

Reproducing kernel Hilbert spaces provide a foundational framework for kernel-based learning, where regularization and interpolation problems admit finite-dimensional solutions through classical representer theorems. Many modern learning models, however – including fixed-architecture neural networks equipped with non-quadratic norms – naturally give rise to non-Hilbertian geometries that fall outside this setting. In Banach spaces, continuity of point-evaluation functionals alone is insufficient to guarantee feature representations or kernel-based learning formulations. In this work, we develop a functional-analytic framework for learning in Banach spaces based on the notion of featured reproducing kernel Banach spaces. We identify the precise structural conditions under which feature maps, kernel constructions, and representer-type results can be recovered beyond the Hilbertian regime. Within this framework, supervised learning is formulated as a minimal-norm interpolation or regularization problem, and existence results together with conditional representer theorems are established. We further extend the theory to vector-valued featured reproducing kernel Banach spaces and show that fixed-architecture neural networks naturally induce special instances of such spaces. This provides a unified function-space perspective on kernel methods and neural networks and clarifies when kernel-based learning principles extend beyond reproducing kernel Hilbert spaces.

💡 Research Summary

The paper addresses a fundamental gap between classical kernel‑based learning, which relies on reproducing kernel Hilbert spaces (RKHS), and modern learning models—particularly fixed‑architecture neural networks equipped with non‑quadratic norms—that naturally live in Banach spaces. While continuity of point‑evaluation functionals guarantees a reproducing kernel in Hilbert spaces, the same is not true in Banach spaces; additional structure is required to obtain feature maps, kernels, and representer‑type results.

To fill this gap the authors introduce featured reproducing kernel Banach spaces (featured RKBS). A Banach space ( \mathcal{B} \subset \mathcal{F}(X,\mathbb{K}) ) is called featured if there exists a Banach “pre‑dual” ( \mathcal{E} ) and a feature map ( \Phi : X \to \mathcal{E} ) such that every function ( f\in\mathcal{B} ) can be written as
( f(x)=\langle f,\Phi(x)\rangle_{\mathcal{E}} ) for all ( x\in X ).
When ( \mathcal{E} ) itself possesses a reproducing kernel ( k_{\mathcal{E}} ), the induced kernel on ( \mathcal{B} ) is simply
( k(x,x’) = \langle \Phi(x),\Phi(x’)\rangle_{\mathcal{E}} ).
Under these structural assumptions the authors prove:

1. Existence of solutions for minimal‑norm interpolation and regularized risk minimization, using Hahn–Banach extension arguments.
2. Conditional representer theorems: if the regularizer is norm‑monotone and the optimization problem can be expressed via sub‑differentials in the dual of ( \mathcal{E} ), any minimizer admits a finite expansion
   ( f(\cdot)=\sum_{i=1}^{n}\alpha_i k(\cdot,x_i) ).
   The theorem is “conditional” because it holds only when the featured structure is present; in general RKBS the solution may remain in a weak‑* closed infinite‑dimensional set.

The paper then defines a special subclass of featured RKBS where the pre‑dual ( \mathcal{E} ) is reflexive and the span of ( \Phi(X) ) is dense in ( \mathcal{E} ). In this case the representer theorem becomes unconditional: every solution to the regularized problem necessarily has the finite kernel expansion above, mirroring the classical RKHS result.

A substantial portion of the work extends the theory to vector‑valued (multi‑output) spaces. By taking a Banach space of vector‑valued functions as the pre‑dual, the authors construct matrix‑valued kernels ( K(x,x’) = \langle \Phi(x),\Phi(x’)\rangle_{\mathcal{E}} ) and prove analogous representer theorems for each output component, thus covering multi‑task and multi‑label learning.

The most novel contribution is the connection to fixed‑architecture neural networks. For a network with depth ( L ), activation ( \sigma ), and a parameter norm ( |\cdot|_p ), the set of functions realizable by the network can be identified with a featured RKBS whose pre‑dual is the tensor product of the layer‑wise Banach spaces induced by the ( \ell_p ) norms. The network’s forward map itself serves as the feature map ( \Phi ). Consequently, training the network with a regularizer that penalizes the parameter norm is equivalent to solving a minimal‑norm interpolation problem in the associated featured RKBS. The representer theorem then tells us that the optimal parameters can be expressed as a finite combination of kernel sections centered at the training points—providing a rigorous function‑space justification for many empirical observations about “implicit bias” in over‑parameterized networks. This perspective is distinct from the Neural Tangent Kernel (NTK) limit, which requires infinite width; here the Banach geometry defined by the norm is sufficient.

An illustrative example with a two‑layer ReLU network demonstrates how to construct ( \Phi ) and the kernel explicitly, and verifies that the learned function indeed follows the finite kernel expansion predicted by the theory.

The authors conclude by emphasizing the foundational nature of their results: they delineate the minimal structural assumptions needed for kernel‑based learning in Banach spaces, provide a complete characterization of representer theorems in this setting, and embed fixed‑architecture neural networks within the same functional‑analytic framework. Limitations include the lack of algorithmic development, statistical generalization bounds, and extensions to stochastic or dropout‑based networks—directions earmarked for future work.