Practical Kernel-Based Reinforcement Learning

Kernel-based reinforcement learning (KBRL) stands out among reinforcement learning algorithms for its strong theoretical guarantees. By casting the learning problem as a local kernel approximation, KBRL provides a way of computing a decision policy which is statistically consistent and converges to a unique solution. Unfortunately, the model constructed by KBRL grows with the number of sample transitions, resulting in a computational cost that precludes its application to large-scale or on-line domains. In this paper we introduce an algorithm that turns KBRL into a practical reinforcement learning tool. Kernel-based stochastic factorization (KBSF) builds on a simple idea: when a transition matrix is represented as the product of two stochastic matrices, one can swap the factors of the multiplication to obtain another transition matrix, potentially much smaller, which retains some fundamental properties of its precursor. KBSF exploits such an insight to compress the information contained in KBRL’s model into an approximator of fixed size. This makes it possible to build an approximation that takes into account both the difficulty of the problem and the associated computational cost. KBSF’s computational complexity is linear in the number of sample transitions, which is the best one can do without discarding data. Moreover, the algorithm’s simple mechanics allow for a fully incremental implementation that makes the amount of memory used independent of the number of sample transitions. The result is a kernel-based reinforcement learning algorithm that can be applied to large-scale problems in both off-line and on-line regimes. We derive upper bounds for the distance between the value functions computed by KBRL and KBSF using the same data. We also illustrate the potential of our algorithm in an extensive empirical study in which KBSF is applied to difficult tasks based on real-world data.

💡 Research Summary

The paper addresses a fundamental scalability bottleneck of Kernel‑Based Reinforcement Learning (KBRL). While KBRL enjoys strong theoretical guarantees—statistical consistency and convergence to a unique optimal solution—its model grows linearly with the number of sampled transitions. Consequently, the Bellman updates on the constructed finite MDP become O(n²|A|) and quickly become infeasible for large‑scale or online problems.

To overcome this, the authors introduce Kernel‑Based Stochastic Factorization (KBSF). The key mathematical insight is that a stochastic transition matrix P can be factorized as P = D K, where D and K are also stochastic matrices of dimensions n × m and m × n respectively (m < n). Interpreting D as probabilities of moving from original states to a set of m artificial “latent” states, and K as probabilities of moving back, one can swap the order of multiplication to obtain a reduced transition matrix (\tilde P = K D) of size m × m. This new matrix preserves important properties of the original dynamics while dramatically shrinking the state space.

KBSF builds on this trick as follows. From a batch of transitions ({(s_i^a, r_i^a, \hat s_i^a)}{i=1}^{n_a}) for each action a, a kernel function φ with bandwidth τ defines normalized kernel weights (\kappa\tau^a(s, s_i^a)). Using these weights, KBRL would construct a large finite MDP with states (\hat s_i^a). KBSF instead selects a fixed number m of artificial states (often chosen as kernel centers) and defines D and K directly from the same kernel weights, ensuring both matrices are row‑stochastic. The product K D yields a compact transition model on the artificial states.

With this compact model, standard dynamic programming (Bellman operator T = ΓΔ) is applied to compute an approximate optimal value function (\tilde V^*). The authors prove two main theoretical results. First, they derive an explicit L∞ bound on the difference between the value function obtained by KBSF and the one obtained by the original KBRL using the same data. The bound depends on τ, the number of artificial states m, the sample size n, and the Lipschitz constants of the underlying MDP. By increasing m (or adjusting τ) the bound can be made arbitrarily small, showing that KBSF can approximate KBRL as closely as desired. Second, they show that the memory requirement of KBSF is O(m²) and independent of n, while the computational cost of constructing the factorization and performing each Bellman update is O(n · m). Hence KBSF achieves linear‑in‑samples time complexity, which is optimal if no data are discarded.

The paper presents both an offline version (all data available beforehand) and an online version. In the online setting, each new transition updates the rows of D and K incrementally; the compact transition matrix is refreshed, and a single Bellman sweep updates the value function. Because the artificial state set remains fixed, memory stays bounded.

Empirical evaluation covers four domains: (1) single pole‑balancing, (2) double pole‑balancing, (3) an HIV drug‑schedule optimization problem, and (4) a real‑world epilepsy suppression task. In offline experiments, KBSF outperforms Least‑Squares Policy Iteration (LSPI) and fitted Q‑iteration, achieving higher cumulative rewards and matching KBRL’s performance when many samples are available. In online experiments, KBSF combined with SARSA learns faster and reaches higher final performance than standard SARSA. Across all tasks, the algorithm’s memory usage remains constant regardless of the number of observed transitions, confirming the theoretical claims.

The discussion examines the impact of deviating from the ideal assumptions (e.g., non‑Lipschitz dynamics, imperfect kernel bandwidth selection) and provides practical guidelines for choosing τ, m, and the kernel centers. Related work is surveyed, positioning KBSF among non‑parametric RL methods, non‑negative matrix factorization techniques, and recent advances in kernel‑based control.

In conclusion, KBSF transforms the elegant but computationally heavy KBRL into a practical tool for both batch and real‑time reinforcement learning. By leveraging stochastic factorization, it decouples the size of the approximating model from the amount of data, delivering linear‑time learning, bounded memory, and provable approximation quality. The approach opens avenues for integrating kernel methods with deep representations, adaptive selection of artificial states, and broader applications in high‑dimensional control problems.