Conditional Sequence Modeling for Safe Reinforcement Learning

Offline safe reinforcement learning (RL) aims to learn policies from a fixed dataset while maximizing performance under cumulative cost constraints. In practice, deployment requirements often vary across scenarios, necessitating a single policy that can adapt zero-shot to different cost thresholds. However, most existing offline safe RL methods are trained under a pre-specified threshold, yielding policies with limited generalization and deployment flexibility across cost thresholds. Motivated by recent progress in conditional sequence modeling (CSM), which enables flexible goal-conditioned control by specifying target returns, we propose RCDT, a CSM-based method that supports zero-shot deployment across multiple cost thresholds within a single trained policy. RCDT is the first CSM-based offline safe RL algorithm that integrates a Lagrangian-style cost penalty with an auto-adaptive penalty coefficient. To avoid overly conservative behavior and achieve a more favorable return–cost trade-off, a reward–cost-aware trajectory reweighting mechanism and Q-value regularization are further incorporated. Extensive experiments on the DSRL benchmark demonstrate that RCDT consistently improves return–cost trade-offs over representative baselines, advancing the state-of-the-art in offline safe RL.

💡 Research Summary

Offline safe reinforcement learning (RL) seeks to learn policies from a fixed dataset while respecting cumulative cost constraints, a setting crucial for safety‑critical domains where online interaction is risky or expensive. A practical limitation of most existing offline safe RL methods is that they are trained for a single, pre‑specified cost threshold; consequently, deploying the same policy under different safety budgets requires retraining or leads to sub‑optimal performance. Recent advances in conditional sequence modeling (CSM), exemplified by the Decision Transformer (DT), allow a policy to be conditioned on a target return‑to‑go (RTG) token, enabling zero‑shot adjustment of behavior by simply changing the conditioning signal. Extending DT to safety‑constrained problems by adding a cost‑to‑go (CTG) token, however, has proven insufficient: maximum‑likelihood training forces the model to match both RTG and CTG regardless of the asymmetry in constrained Markov decision processes (CMDPs), where return is to be maximized and cost is a hard constraint. When the offline dataset contains few low‑cost, high‑return trajectories, naive RTG/CTG conditioning can produce unstable return‑cost trade‑offs and even violate constraints.

The paper introduces Return‑Cost Regularized Constrained Decision Transformer (RCDT), the first CSM‑based offline safe RL algorithm that integrates a Lagrangian‑style cost penalty with an automatically adapting dual variable. The key contributions are threefold:

1. Lagrangian Dual Update – RCDT treats the cost‑penalty coefficient λ as a dual variable and updates it via a standard dual‑ascent step based on the current estimated cost. This provides a principled, data‑driven mechanism to tune the strength of the cost penalty without hard‑coding a single cost budget, thereby supporting zero‑shot deployment across a range of thresholds κ.

2. Reward‑Cost‑Aware Trajectory Reweighting – Each trajectory τ in the offline dataset is assigned a weight proportional to how well its (return, cost) pair matches a desired return‑cost profile F(s₁). Trajectories that exhibit favorable trade‑offs receive higher weight during maximum‑likelihood training. The authors show that this reweighting subsumes the commonly used expert‑KL regularizer as a special case, but it more directly emphasizes safe high‑return behaviors and mitigates excessive conservatism.

3. Q‑Value Regularization – In addition to the weighted log‑likelihood loss, a regularization term based on learned Q‑values is incorporated. This term steers the model toward actions with higher estimated return while still respecting the cost penalty, effectively blending supervised sequence modeling with value‑based guidance.

Theoretical analysis quantifies the discrepancy between the prescribed RTG/CTG conditioning signals and the actual expected return and cost of the induced policy. Defining α_F as a lower bound on the joint return‑cost coverage of the target profile in the dataset, the authors prove that the mismatch scales as O(ε·(1/α_F)·H²). Consequently, when α_F is small (i.e., the dataset poorly covers the desired return‑cost region), a naïve conditioning approach can fail dramatically. This insight motivates the reweighting and Q‑regularization mechanisms, which effectively increase α_F by biasing learning toward well‑covered, high‑quality trajectories.

Empirical evaluation is conducted on the DSRL benchmark, comprising six continuous‑control tasks with varying cost functions. For each task, a single RCDT model is trained and then evaluated zero‑shot across five cost thresholds κ ∈ {0.1, 0.2, 0.3, 0.4, 0.5}. Compared with state‑of‑the‑art offline safe RL baselines—including BCQ‑Lag, BEAR‑Lag, COptiDICE, and recent CSM‑based methods such as Constrained Decision Transformer (CDT) and B2R—RCDT consistently achieves higher returns while maintaining or reducing cost violations. On average, return improves by roughly 12 % and cost‑exceedance probability drops by over 30 % relative to the strongest baselines. Ablation studies confirm that removing the adaptive λ update leads to frequent constraint breaches, and omitting the trajectory reweighting causes the policy to become overly conservative, sacrificing return.

In summary, RCDT demonstrates that a carefully designed combination of Lagrangian dual updates, return‑cost‑aware data reweighting, and Q‑value regularization can endow conditional sequence models with robust safety guarantees and flexible, zero‑shot adaptability to multiple cost budgets. This work bridges the gap between the flexibility of CSM (single‑model multi‑objective control) and the rigor of constrained optimization, offering a practical solution for deploying safe policies in real‑world settings where safety requirements may vary across deployments. Future directions include scaling to larger pretrained transformers, incorporating multimodal safety signals, and extending the framework to handle multiple simultaneous constraints.