On the Equilibrium between Feasible Zone and Uncertain Model in Safe Exploration

Ensuring the safety of environmental exploration is a critical problem in reinforcement learning (RL). While limiting exploration to a feasible zone has become widely accepted as a way to ensure safety, key questions remain unresolved: what is the maximum feasible zone achievable through exploration, and how can it be identified? This paper, for the first time, answers these questions by revealing that the goal of safe exploration is to find the equilibrium between the feasible zone and the environment model. This conclusion is based on the understanding that these two components are interdependent: a larger feasible zone leads to a more accurate environment model, and a more accurate model, in turn, enables exploring a larger zone. We propose the first equilibrium-oriented safe exploration framework called safe equilibrium exploration (SEE), which alternates between finding the maximum feasible zone and the least uncertain model. Using a graph formulation of the uncertain model, we prove that the uncertain model obtained by SEE is monotonically refined, the feasible zones monotonically expand, and both converge to the equilibrium of safe exploration. Experiments on classic control tasks show that our algorithm successfully expands the feasible zones with zero constraint violation, and achieves the equilibrium of safe exploration within a few iterations.

💡 Research Summary

The paper tackles a fundamental problem in safe reinforcement learning: how to expand the set of state‑action pairs that can be explored without ever violating safety constraints. Existing approaches either rely on hand‑crafted constraints (e.g., control barrier functions) that are overly conservative, or they learn a model of the environment and gradually enlarge a “safe set” but without a clear theoretical target for how large this set can become.

The authors introduce two tightly coupled concepts. An uncertain model ˆf maps each state‑action pair (x,u) to a set of possible next states, capturing model error as a bounded set rather than a probabilistic distribution. If the true dynamics f(x,u) always lies inside this set, the model is said to be well‑calibrated and safety can be guaranteed by requiring that every element of the set satisfies the constraints. A feasible zone Z ⊆ X×U is defined as a collection of state‑action pairs that are both constraint‑satisfying and forward‑invariant under the uncertain model: starting from any (x,u)∈Z, all possible next states remain within Z.

Crucially, the size of Z and the accuracy of ˆf are mutually dependent. A larger feasible zone provides more diverse data, which can be used to shrink the transition sets of ˆf (i.e., reduce model uncertainty). Conversely, a more accurate model yields smaller transition sets, allowing more (x,u) pairs to be certified as safe, thus expanding Z. This feedback loop suggests that the ultimate goal of safe exploration is to reach an equilibrium where Z is the maximum feasible zone for a given model, and ˆf is the least uncertain model for that zone. At equilibrium neither the zone can be expanded nor the model can be refined further.

To operationalize this insight, the authors propose Safe Equilibrium Exploration (SEE). SEE alternates between two steps:

1. Maximum feasible zone computation – Given the current uncertain model, the algorithm solves a “risky Bellman equation” via fixed‑point iteration to obtain the largest forward‑invariant set Z*. This step guarantees that any policy confined to Z* will never violate constraints under the worst‑case transition in ˆf.

2. Least uncertain model update – Using data collected only inside Z*, the algorithm refines the transition sets. The refinement problem is cast as a graph problem: each transition pair is a node, and edges encode incompatibility under the Lipschitz‑based uncertainty bound. Deciding which transition pairs can be safely removed is shown to be equivalent to the NP‑hard clique decision problem. The authors therefore devise a polynomial‑time approximation based on a sufficient condition for removability, which in practice yields a substantially tighter model.

The paper proves that each iteration of SEE monotonically expands the feasible zone and monotonically reduces model uncertainty. Consequently, the sequence converges to the equilibrium described above.

Empirical evaluation on classic control benchmarks (Pendulum, CartPole, MountainCar) demonstrates that SEE rapidly expands the safe region while maintaining zero constraint violations. Compared with safety‑filter baselines that rely on fixed barrier functions, SEE achieves a far larger safe region after only a few iterations, confirming the practical value of the equilibrium perspective.

In summary, the contribution of the work is threefold: (1) introducing the least uncertain model concept that links model learning directly to safe set expansion, (2) formalizing safe exploration as a joint optimization problem whose solution is an equilibrium between feasible zone and model, and (3) providing a concrete algorithm with theoretical guarantees and empirical validation. The approach opens new avenues for safe RL in real‑world systems where both model uncertainty and safety constraints must be handled simultaneously.