Reward-Preserving Attacks For Robust Reinforcement Learning

Adversarial training in reinforcement learning (RL) is challenging because perturbations cascade through trajectories and compound over time, making fixed-strength attacks either overly destructive or too conservative. We propose reward-preserving attacks, which adapt adversarial strength so that an $α$ fraction of the nominal-to-worst-case return gap remains achievable at each state. In deep RL, perturbation magnitudes $η$ are selected dynamically, using a learned critic $Q((s,a),η)$ that estimates the expected return of $α$-reward-preserving rollouts. For intermediate values of $α$, this adaptive training yields policies that are robust across a wide range of perturbation magnitudes while preserving nominal performance, outperforming fixed-radius and uniformly sampled-radius adversarial training.

💡 Research Summary

This paper tackles a fundamental difficulty in adversarial training for reinforcement learning (RL): perturbations accumulate over time, making a fixed‑strength attack either too destructive—preventing learning or even rendering the task unsolvable—or too weak to induce meaningful robustness. To address this, the authors introduce “reward‑preserving attacks,” a novel class of adaptive adversarial perturbations that guarantee a user‑specified fraction α of the nominal‑to‑worst‑case return gap remains achievable at every state‑action pair.

Formally, for a given state s and action a, let QΩ(s,a) be the optimal Q‑value in the nominal MDP Ω and QΩ,ξ*(s,a) the optimal Q‑value under the worst‑case attack ξ* within an uncertainty set B. The gap Δ(s,a)=QΩ(s,a)−QΩ,ξ*(s,a) quantifies how much return is lost under the strongest possible attack. An attack ξ is defined as α‑reward‑preserving if it ensures QΩ,ξ(s,a) ≥ QΩ,ξ*(s,a)+α·Δ(s,a). Thus, α=0 corresponds to pure worst‑case robustness, while α=1 preserves the full nominal return; intermediate α values balance the two extremes.

In tabular MDPs, directly interpolating between the nominal and worst‑case Q‑values (i.e., ˆQ(s,a)=QΩ,ξ(s,a)+α·Δ(s,a)) does not satisfy the Bellman optimality equations, because the set of admissible attacks depends on the optimal policy adapted to those attacks. The authors therefore propose a two‑stage iterative scheme: (1) for a fixed policy π, find the worst‑case α‑reward‑preserving attack ξ∈Ξα(s,a) that minimizes Qπ,Ωξ(s,a); (2) update the policy against this attack using standard RL updates. Repeating these steps yields convergence to a policy that respects the α‑preserving constraint while remaining Bellman‑consistent.

For deep RL, the attack magnitude η is treated as a continuous scalar. A critic network Q((s,a),η) is trained to predict the expected return of an α‑reward‑preserving rollout given a candidate η. During training, η is adjusted via gradient descent to the smallest value that still satisfies the α‑preserving inequality for the current state. Consequently, the perturbation strength becomes state‑dependent: in “dangerous” regions (e.g., a narrow bridge in a gridworld) η is reduced, whereas in safe regions η can be larger. This dynamic scaling prevents the agent from being trapped by overly strong perturbations while still exposing it to meaningful distribution shifts.

Two theoretical properties are highlighted. First, when the uncertainty set B is sufficiently large so that the worst‑case attack collapses all Q‑values to a constant minimum reward Rmin, any α‑reward‑preserving attack preserves the ordering of Q‑values. Hence the optimal policy for the perturbed MDP coincides with the nominal optimal policy, guaranteeing that the agent can still recover from severe attacks without altering its intended behavior. Second, the authors derive a “preference reversal condition” showing how α controls the trade‑off between nominal performance and worst‑case robustness: for α<0.5 the agent favors safer actions even if they sacrifice nominal reward, while for α>0.5 the nominal reward dominates and the policy remains close to the original optimal one.

Empirical evaluation spans three domains: (i) a deterministic gridworld with a narrow bridge illustrating a critical region; (ii) continuous control tasks from the MuJoCo suite; and (iii) several Atari games. Baselines include fixed‑radius L₂/L∞ attacks and uniformly sampled‑radius attacks. Results consistently demonstrate that intermediate α values (≈0.3–0.5) yield policies that are robust across a wide spectrum of perturbation magnitudes while incurring negligible loss in nominal performance. In the gridworld, fixed‑radius attacks either make the bridge impassable or leave the agent unchallenged; the reward‑preserving approach automatically weakens the attack on the bridge, allowing the agent to learn the optimal path. In MuJoCo and Atari, the proposed method outperforms baselines in both worst‑case return and nominal return, confirming that dynamic, state‑aware perturbation scaling leads to superior robustness without sacrificing efficiency.

The paper concludes by acknowledging limitations and future directions. Currently α is a static hyper‑parameter; an adaptive scheme that tunes α online could further improve performance. Extending the framework to multi‑agent settings, handling partial observability more explicitly, and reducing the computational overhead of the critic network for real‑time robotic applications are identified as promising avenues for continued research. Overall, reward‑preserving attacks provide a principled and practical mechanism to balance safety and performance in adversarially trained reinforcement learners.