A Review On Safe Reinforcement Learning Using Lyapunov and Barrier Functions

Reinforcement learning (RL) has proven to be particularly effective in solving complex decision-making problems for a wide range of applications. From a control theory perspective, RL can be considered as an adaptive optimal control scheme. Lyapunov and barrier functions are the most commonly used certificates to guarantee system stability for a proposed/derived controller and constraint satisfaction guarantees, respectively, in control theoretic approaches. However, compared to theoretical guarantees available in control theoretic methods, RL lacks closed-loop stability of a computed policy and constraint satisfaction guarantees. Safe reinforcement learning refers to a class of constrained problems where the constraint violations lead to partial or complete system failure. The goal of this review is to provide an overview of safe RL techniques using Lyapunov and barrier functions to guarantee this notion of safety discussed (stability of the system in terms of a computed policy and constraint satisfaction during training and deployment). The different approaches employed are discussed in detail along with their shortcomings and benefits to provide critique and possible future research directions. Key motivation for this review is to discuss current theoretical approaches for safety and stability guarantees in RL similar to control theoretic approaches using Lyapunov and barrier functions. The review provides proven potential and promising scope of providing safety guarantees for complex dynamical systems with operational constraints using model-based and model-free RL.

💡 Research Summary

This paper presents a comprehensive review of safe reinforcement learning (RL) techniques that employ Lyapunov functions and barrier functions to provide stability and constraint‑satisfaction guarantees, mirroring the certification mechanisms long used in control theory. The authors begin by positioning RL as an adaptive optimal control scheme and highlighting the gap between the strong theoretical guarantees available for model‑based control (e.g., Lyapunov stability, barrier‑based safety sets) and the relatively weak guarantees typically offered by standard RL algorithms. They argue that for safety‑critical applications—where constraint violations can cause catastrophic failure—bridging this gap is essential.

The review is organized around four major methodological families.

1. Lyapunov‑Based Safe RL – The authors discuss how control Lyapunov functions (CLFs) can be used to certify that a learned policy drives the closed‑loop system toward a desired equilibrium. In model‑based settings, the Lie derivative condition ∇V·f(x)+∇V·g(x)u ≤ 0 is enforced via a quadratic program (QP) that yields a stabilizing control input. For model‑free approaches, recent works embed Lyapunov constraints into the loss functions of neural‑network value approximators or project the policy onto a Lyapunov‑admissible set after each gradient update. The review highlights both the theoretical soundness of this approach and the practical difficulty of constructing a suitable Lyapunov function for high‑dimensional, nonlinear systems.

2. Barrier‑Based Safe RL – Control barrier functions (CBFs) define a safe set C and guarantee forward invariance by enforcing L_f h(x)+L_g h(x)u+α(h(x)) ≥ 0, again typically solved as a QP. The paper surveys works that combine CBFs with RL in three ways: (i) as a hard safety filter that overrides unsafe actions, (ii) as a soft penalty added to the RL reward, and (iii) as part of a joint CLF‑CBF QP that simultaneously ensures stability and safety. The authors note that CBFs are particularly attractive for online safety because they provide explicit, state‑dependent constraints that can be evaluated in real time.

3. Filtering, Shielding, and Human‑in‑the‑Loop – Safety filters (confidence‑based, convex‑optimization based) and shielding mechanisms (centralized or factored shields derived from linear temporal logic) are examined. These methods pre‑emptively block actions that would violate a safety specification, often by consulting a backup policy or a formally synthesized shield. Human supervision, sometimes called “RL via shielding,” is also covered, emphasizing how expert demonstrations or corrective feedback can be used to penalize unsafe behavior during training. The main limitation identified is the reliance on prior knowledge of the system dynamics or safety specifications, which may be unavailable in many real‑world scenarios.

4. Integration with Model Predictive Control (MPC) and Quadratic Programming (QP) – The review details several hybrid schemes where RL provides exploratory or performance‑enhancing components while MPC supplies a robust, constraint‑aware backbone. Notable examples include using online data to update MPC cost matrices, approximating value functions with MPC‑derived QPs, and alternating between RL policy updates and MPC feasibility checks. These approaches inherit the interpretability and constraint‑handling strengths of MPC, but they also inherit its computational burden, especially when embedded in high‑frequency control loops.

After cataloguing these approaches, the authors critically assess common shortcomings: (a) the design of Lyapunov and barrier certificates is highly problem‑specific and lacks systematic, automated procedures; (b) real‑time solution of QPs or MPC problems can be prohibitive for large‑scale or fast‑dynamics systems; (c) the trade‑off between safety (conservatism) and exploration efficiency is often handled heuristically rather than through rigorous analysis; (d) multi‑agent settings pose additional challenges for coordinated safety guarantees, and existing literature offers limited solutions.

The paper concludes with a forward‑looking research agenda: (1) Automatic certificate synthesis via meta‑learning, neural‑network Lyapunov/barrier approximators, or data‑driven symbolic regression; (2) Lightweight real‑time optimization through approximate QP solvers, distributed computation, or learning‑based warm‑starts; (3) Quantitative safety‑exploration trade‑off analysis using constrained Bayesian optimization, Lagrangian dual methods, or risk‑sensitive RL formulations; and (4) Scalable multi‑agent safety through decentralized CLF/CBF design, consensus‑based shielding, and communication‑aware safety protocols.

Overall, the review convincingly demonstrates that integrating Lyapunov and barrier function theory into RL yields a promising pathway toward provably safe, stable learning‑based controllers for complex dynamical systems. By systematically summarizing existing methods, exposing their limitations, and outlining concrete research directions, the paper serves as a valuable roadmap for both theoreticians and practitioners aiming to bring safe RL from simulation to real‑world deployment.