Efficient Algorithms for Robust Markov Decision Processes with $s$-Rectangular Ambiguity Sets

Robust Markov decision processes (MDPs) have attracted significant interest due to their ability to protect MDPs from poor out-of-sample performance in the presence of ambiguity. In contrast to classical MDPs, which account for stochasticity by modeling the dynamics through a stochastic process with a known transition kernel, a robust MDP additionally accounts for ambiguity by optimizing against the most adverse transition kernel from an ambiguity set constructed via historical data. In this paper, we develop a unified solution framework for a broad class of robust MDPs with $s$-rectangular ambiguity sets, where the most adverse transition probabilities are considered independently for each state. Using our algorithms, we show that $s$-rectangular robust MDPs with $1$- and $2$-norm as well as $ϕ$-divergence ambiguity sets can be solved several orders of magnitude faster than with state-of-the-art commercial solvers, and often only a logarithmic factor slower than classical MDPs. We demonstrate the favorable scaling properties of our algorithms on a range of synthetically generated as well as standard benchmark instances.

💡 Research Summary

The paper addresses the computational challenge of solving robust Markov decision processes (MDPs) when the uncertainty in transition probabilities is modeled by s‑rectangular ambiguity sets. Unlike the more restrictive (s,a)‑rectangular sets, s‑rectangular sets impose a single budget per state on the deviation of all action‑specific transition distributions from nominal estimates, allowing the adversary to couple actions within the same state. This structure preserves the dynamic‑programming (DP) decomposition (Bellman optimality) while potentially yielding less conservative policies, but it also permits optimal policies to be randomized, complicating solution methods.

The authors first show that for any s‑rectangular set of the form

\