Continuous-time reinforcement learning: ellipticity enables model-free value function approximation

We study off-policy reinforcement learning for controlling continuous-time Markov diffusion processes with discrete-time observations and actions. We consider model-free algorithms with function approximation that learn value and advantage functions directly from data, without unrealistic structural assumptions on the dynamics. Leveraging the ellipticity of the diffusions, we establish a new class of Hilbert-space positive definiteness and boundedness properties for the Bellman operators. Based on these properties, we propose the Sobolev-prox fitted $q$-learning algorithm, which learns value and advantage functions by iteratively solving least-squares regression problems. We derive oracle inequalities for the estimation error, governed by (i) the best approximation error of the function classes, (ii) their localized complexity, (iii) exponentially decaying optimization error, and (iv) numerical discretization error. These results identify ellipticity as a key structural property that renders reinforcement learning with function approximation for Markov diffusions no harder than supervised learning.

💡 Research Summary

This paper addresses off‑policy reinforcement learning (RL) for continuous‑time controlled diffusion processes observed and acted upon at discrete time steps. The authors consider a stochastic differential equation (SDE) dXₜ = b₍ₐ₎(Xₜ) dt + Λ(Xₜ)^{1/2} dBₜ with unknown drift b and diffusion matrix Λ, a bounded reward function r, and a discount factor β > 0. By sampling the process every η > 0 units of time, the continuous‑time problem is cast as a discrete‑time Markov decision process (MDP) with transition kernel Pₐ^η, discount e^{−βη}, and per‑step reward η r. Data are collected offline under a fixed behavior policy π₀, yielding n independent trajectories that terminate at a geometrically distributed random horizon (rate β).

A central difficulty in RL with function approximation is that the Bellman operator is a contraction only under the supremum norm, not under the L²‑norm induced by the data distribution. This mismatch can cause instability for fitted‑Q, TD, and related algorithms. Existing remedies typically impose strong structural assumptions such as Bellman completeness, uniform coverage of the state‑action space, or linear‑quadratic dynamics—assumptions that are hard to verify in practice.

The key insight of the paper is to exploit uniform ellipticity of the diffusion matrix: there exists a constant c > 0 such that c I ≼ Λ(x) ≼ C I for all states x. Under this condition the generator Aₐ of the diffusion is positive‑definite and bounded in Sobolev space H¹(ρ) for any suitable reference measure ρ. Consequently, the Bellman operator becomes a non‑expansive mapping in the H¹‑norm, enabling a well‑behaved projected Bellman equation.

Building on these operator properties, the authors propose the Sobolev‑prox fitted q‑learning algorithm. Two function classes are fixed in advance: F_v ⊂ H¹ for the value function and F_q ⊂ H¹ for the advantage (or Q‑) function. Each iteration consists of:

1. Advantage update – solve a least‑squares regression over F_q using the Bellman residual R + e^{−βη} maxₐ′ v_k(X′) − q(X,A).
2. Value update – perform a proximal step in the Sobolev norm, projecting the result onto F_v.

Both steps reduce to standard regression and projection operations, so the computational cost matches that of supervised learning in the respective function classes.

The main theoretical contribution is an oracle inequality for the algorithm’s output (v̂, q̂). The error bound decomposes into four terms:

(i) the best approximation errors of v* and q* by the chosen function classes (measured in H¹);
(ii) a critical radius determined by the localized complexities (covering numbers) of F_v and F_q, which governs the statistical rate;
(iii) an exponentially decaying optimization error reflecting the number of proximal‑gradient iterations; and
(iv) a discretization bias scaling as η^{1/2}, arising from the continuous‑to‑discrete time conversion.

Importantly, the bound does not require Bellman completeness, uniform visitation, or linear dynamics. The ellipticity assumption alone guarantees that the projected Bellman equation is well‑posed and that learning is no harder than supervised regression. The statistical rates match optimal supervised learning rates up to logarithmic factors, and the algorithm’s sample complexity is essentially O(1/√n) for typical smooth function classes.

The paper situates its contributions within a broad literature: continuous‑time RL (martingale‑based methods, diffusion‑generative‑model fine‑tuning), RL with function approximation (linear and non‑linear settings, recent attempts to bridge RL and supervised learning), and statistical learning for elliptic PDEs. By focusing on the elliptic structure of the Hamilton–Jacobi–Bellman equation associated with the diffusion control problem, the authors provide a novel bridge between stochastic control theory and modern statistical learning.

In conclusion, the work demonstrates that uniform ellipticity is a powerful structural property that renders model‑free RL with function approximation for diffusion processes as tractable as supervised learning. The Sobolev‑prox fitted q‑learning algorithm offers a practical, provably stable method that can be applied to finance, robotics, queueing networks, and reward‑guided fine‑tuning of diffusion generative models. Future directions include extending the analysis to non‑elliptic dynamics, handling control constraints, and integrating adaptive deep function approximators for F_v and F_q.