Certifying Hamilton-Jacobi Reachability Learned via Reinforcement Learning

We present a framework to \emph{certify} Hamilton–Jacobi (HJ) reachability learned by reinforcement learning (RL). Building on a discounted initial time \emph{travel-cost} formulation that makes small-step RL value iteration provably equivalent to a forward Hamilton–Jacobi (HJ) equation with damping, we convert certified learning errors into calibrated inner/outer enclosures of strict backward reachable tube. The core device is an additive-offset identity: if $W_λ$ solves the discounted travel-cost Hamilton–Jacobi–Bellman (HJB) equation, then $W_\varepsilon:=W_λ+ \varepsilon$ solves the same PDE with a constant offset $λ\varepsilon$. This means that a uniform value error is \emph{exactly} equal to a constant HJB offset. We establish this uniform value error via two routes: (A) a Bellman operator-residual bound, and (B) a HJB PDE-slack bound. Our framework preserves HJ-level safety semantics and is compatible with deep RL. We demonstrate the approach on a double-integrator system by formally certifying, via satisfiability modulo theories (SMT), a value function learned through reinforcement learning to induce provably correct inner and outer backward-reachable set enclosures over a compact region of interest.

💡 Research Summary

The paper introduces a novel certification framework that bridges reinforcement learning (RL)–derived value functions and Hamilton‑Jacobi (HJ) reachability analysis, enabling provable safety guarantees for systems learned via RL. The authors start by formulating a discounted “travel‑cost” optimal control problem, where the cost of reaching a target from an initial state is accumulated with an exponential discount factor λ. They prove that performing small‑step value iteration on this problem is mathematically equivalent to solving a forward HJ partial differential equation (PDE) that includes a damping term proportional to λ. This equivalence establishes a direct link between the Bellman updates used in RL and the Hamilton‑Jacobi‑Bellman (HJB) equation governing reachability.

The central technical contribution is the “additive‑offset identity.” If a function (W_{\lambda}) satisfies the discounted travel‑cost HJB equation
\