Viability of Future Actions: Robust Safety in Reinforcement Learning via Entropy Regularization

Despite the many recent advances in reinforcement learning (RL), the question of learning policies that robustly satisfy state constraints under unknown disturbances remains open. In this paper, we offer a new perspective on achieving robust safety by analyzing the interplay between two well-established techniques in model-free RL: entropy regularization, and constraints penalization. We reveal empirically that entropy regularization in constrained RL inherently biases learning toward maximizing the number of future viable actions, thereby promoting constraints satisfaction robust to action noise. Furthermore, we show that by relaxing strict safety constraints through penalties, the constrained RL problem can be approximated arbitrarily closely by an unconstrained one and thus solved using standard model-free RL. This reformulation preserves both safety and optimality while empirically improving resilience to disturbances. Our results indicate that the connection between entropy regularization and robustness is a promising avenue for further empirical and theoretical investigation, as it enables robust safety in RL through simple reward shaping.

💡 Research Summary

The paper tackles the long‑standing problem of learning reinforcement‑learning (RL) policies that satisfy hard state constraints even when the system is subject to unknown disturbances. The authors propose a fresh perspective: by combining two widely used model‑free RL techniques—maximum‑entropy regularization and penalty‑based constraint relaxation—they can obtain policies that are both safe and robust to action noise without resorting to specialized robust‑RL algorithms.

The core insight is that entropy regularization implicitly encourages a policy to stay in states that offer a larger set of viable future actions. In the authors’ terminology, the cumulative discounted entropy of a policy can be interpreted as a proxy for the expected number of safe actions available in the future. Consequently, when the temperature parameter α (which scales the entropy term) is increased, the policy’s mode (the most likely trajectory) is “repelled” from constraint boundaries, preferring regions where many actions keep the system inside the viability kernel. This behavior is illustrated on a “fenced‑cliff” grid world: higher α values produce trajectories that keep a larger safety margin from the fence, at the cost of longer travel time.

The second contribution is a theoretical analysis of penalty methods in the presence of entropy regularization. Classical results (e.g., from