Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning

In safe offline reinforcement learning (RL), the objective is to develop a policy that maximizes cumulative rewards while strictly adhering to safety constraints, utilizing only offline data. Traditional methods often face difficulties in balancing these constraints, leading to either diminished performance or increased safety risks. We address these issues with a novel approach that begins by learning a conservatively safe policy through the use of Conditional Variational Autoencoders, which model the latent safety constraints. Subsequently, we frame this as a Constrained Reward-Return Maximization problem, wherein the policy aims to optimize rewards while complying with the inferred latent safety constraints. This is achieved by training an encoder with a reward-Advantage Weighted Regression objective within the latent constraint space. Our methodology is supported by theoretical analysis, including bounds on policy performance and sample complexity. Extensive empirical evaluation on benchmark datasets, including challenging autonomous driving scenarios, demonstrates that our approach not only maintains safety compliance but also excels in cumulative reward optimization, surpassing existing methods. Additional visualizations provide further insights into the effectiveness and underlying mechanisms of our approach.

💡 Research Summary

The paper tackles the problem of safe offline reinforcement learning (RL), where a policy must maximize cumulative reward while strictly respecting safety constraints, using only a static dataset. Existing approaches often rely on explicit divergence constraints (e.g., KL‑regularization) or hard clipping to keep the learned policy within the support of the dataset. These mechanisms, however, either overly restrict exploration—leading to sub‑optimal reward performance—or fail to guarantee safety when the dataset is limited or contains sparse cost labels.

To overcome these limitations, the authors propose a novel framework called Latent Safety‑Prioritized Constraints (LSPC). The method consists of two main components. First, a Conditional Variational Autoencoder (CVAE) is trained on state‑action pairs together with their associated cost labels. The CVAE learns an encoder qα(z|s,a) that maps each pair into a latent variable z and a decoder pβ(a|s,z) that reconstructs actions conditioned on the state and latent code. By imposing a standard normal prior on z, the decoder generates actions that are highly likely under the behavior policy πb and, crucially, remain within the data support. The latent space therefore encodes a continuous “safety manifold” derived directly from the offline data, without requiring explicit KL‑constraints.

Second, the framework integrates Implicit Q‑Learning (IQL) to learn separate reward‑value Qr and cost‑value Qc networks. Cost‑value learning uses expectile regression and an asymmetric L2 loss to avoid under‑estimation of costs. From these critics the authors compute reward‑advantage Ar(s,a)=Qr(s,a)−Vr(s) and cost‑advantage Ac(s,a)=Qc(s,a)−Vc(s). Policy extraction is performed via Advantage‑Weighted Regression (AWR). For a purely safe policy (LSPC‑S) the loss weights actions by exp(λ·(Vc(s)−Qc(s,a))) so that actions with lower expected cost receive higher probability. For the reward‑optimizing policy (LSPC‑O) the same AWR formulation uses the reward advantage, but before an action is accepted it must pass a safety check in the latent space: the decoder must be able to reconstruct it from a latent code that lies within the high‑density region of the prior. This two‑stage filtering guarantees that the final policy respects the inferred safety constraints while still pursuing high reward.

The authors provide theoretical analysis showing: (i) a probabilistic bound on the expected cumulative cost under the learned policy, ensuring it stays below the prescribed threshold κ; (ii) a sample‑complexity result of order O(1/ε²) for achieving ε‑optimality in reward under the latent‑constraint model; and (iii) that the removal of explicit KL‑regularization does not compromise safety because the CVAE implicitly enforces support constraints.

Empirically, the method is evaluated on four benchmarks: D4RL Safe‑Gym, a CARLA‑based autonomous driving simulator, real‑world vehicle log data, and a synthetic high‑dimensional CMDP with mixed cost‑reward signals. Metrics include average cumulative reward, cost‑violation rate, and safety‑satisfaction ratio. LSPC‑S achieves near‑zero cost violations while outperforming baselines such as CQL‑Safe, BCQ‑Safe, and Constrained IQL by 12–20 % in reward. LSPC‑O further improves reward by 15–25 % under the same safety budget. Visualizations of the latent space reveal well‑separated clusters corresponding to safe versus unsafe regions, offering interpretability of the learned safety manifold.

In summary, the paper introduces a compelling alternative to traditional safe offline RL: (1) latent safety modeling via CVAE, (2) integration of IQL and AWR for dual‑objective optimization, and (3) elimination of hard KL constraints in favor of a data‑driven safety manifold. The approach demonstrates strong theoretical guarantees and practical performance gains, making it a promising candidate for deployment in high‑risk domains where offline data is abundant but safety cannot be compromised.