Best-of-Both-Worlds for Heavy-Tailed Markov Decision Processes

We investigate episodic Markov Decision Processes with heavy-tailed feedback (HTMDPs). Existing approaches for HTMDPs are conservative in stochastic environments and lack adaptivity in adversarial regimes. In this work, we propose algorithms HT-FTRL-OM and HT-FTRL-UOB for HTMDPs that achieve Best-of-Both-Worlds (BoBW) guarantees: instance-independent regret in adversarial environments and logarithmic instance-dependent regret in self-bounding (including the stochastic case) environments. For the known transition setting, HT-FTRL-OM applies the Follow-The-Regularized-Leader (FTRL) framework over occupancy measures with novel skipping loss estimators, achieving a $\widetilde{O}(T^{1/α})$ regret bound in adversarial regimes and a $O(\log T)$ regret in stochastic regimes. Building upon this framework, we develop a novel algorithm HT-FTRL-UOB to tackle the more challenging unknown-transition setting. This algorithm employs a pessimistic skipping loss estimator and achieves a $\widetilde{O}(T^{1/α} + \sqrt{T})$ regret in adversarial regimes and a $O(\log^2(T))$ regret in stochastic regimes. Our analysis overcomes key barriers through several technical insights, including a local control mechanism for heavy-tailed shifted losses, a new suboptimal-mass propagation principle, and a novel regret decomposition that isolates transition uncertainty from heavy-tailed estimation errors and skipping bias.

💡 Research Summary

This paper tackles episodic Markov Decision Processes (MDPs) in which the per‑step loss may follow a heavy‑tailed distribution, i.e., only an α‑moment (1 < α ≤ 2) is assumed to be bounded. Existing work on heavy‑tailed MDPs has focused almost exclusively on stochastic environments and provides only worst‑case (instance‑independent) guarantees; none of them adapt to adversarial settings nor achieve the logarithmic instance‑dependent regret that is standard in stochastic RL. The authors therefore ask whether a Best‑of‑Both‑Worlds (BoBW) algorithm—one that simultaneously attains optimal instance‑independent regret in adversarial regimes and logarithmic instance‑dependent regret in stochastic (or more generally self‑bounding) regimes—can be designed for heavy‑tailed MDPs.

Main contributions

1. HT‑FTRL‑OM (known transition) – The algorithm runs Follow‑the‑Regularized‑Leader (FTRL) over the occupancy‑measure polytope Q(P) using a 1/α‑Tsallis entropy regularizer. A novel “skipping” loss estimator truncates any loss whose magnitude exceeds a data‑dependent threshold τₜ(s,a)=C·σ·t^{1/α}·xₜ(s,a)^{1/α}. The estimator is unbiased after adding a bias‑correction term b^{skip}_t(s,a)=C^{1‑α}·σ·t^{1/α‑1}·xₜ(s,a)^{1/α‑1}. This construction yields a robust estimate even when the loss variance is infinite.
   Regret results: In fully adversarial environments the algorithm achieves (\tilde O(T^{1/α})) regret, matching the minimax rate for heavy‑tailed bandits and improving over prior MDP bounds that contain an extra √T term. In self‑bounding (including stochastic) environments the algorithm enjoys an O(log T) regret, thanks to a new “sub‑optimal mass propagation” principle that shows the occupancy mass on sub‑optimal state‑action pairs shrinks geometrically with the cumulative gap.

2. HT‑FTRL‑UOB (unknown transition) – When the transition kernel is not known, the authors extend the previous framework in two ways. First, they introduce a pessimistic skipping estimator that incorporates an upper‑occupancy bound; this effectively enlarges the feasible occupancy set to a conservative polytope (\bar Q) that contains all occupancy measures consistent with the observed data. Second, they develop a fresh regret decomposition that isolates three sources of error: (i) loss‑estimation error, (ii) transition‑uncertainty error, and (iii) skipping‑bias error. By bounding each term separately, they obtain (\tilde O(T^{1/α}+√T)) regret in adversarial regimes (the √T term stems from the transition‑uncertainty component) and O(log² T) regret in self‑bounding regimes (the extra log factor is due to the pessimistic occupancy bound).

3. Technical innovations –
   Shifted‑loss local control: By centering the loss with the bias‑correction term, the authors obtain concentration for heavy‑tailed variables using only the α‑moment.
   Sub‑optimal mass propagation: A novel principle that links the deviation of occupancy measures across layers of the MDP, enabling a clean logarithmic bound in stochastic settings.
   Regret decomposition with transition isolation: This separates transition estimation errors from heavy‑tailed estimation errors, allowing the two to be handled with different tools (optimistic concentration for losses, pessimistic confidence sets for transitions).
   1/α‑Tsallis entropy: Provides a Bregman divergence that is naturally adapted to α‑moment bounded losses, avoiding the need for boundedness or sub‑Gaussian tails.

Significance
The work demonstrates that BoBW guarantees—previously established only for bounded‑loss MDPs or heavy‑tailed bandits—extend to the much richer setting of heavy‑tailed MDPs with unknown dynamics. The algorithms are computationally tractable (the FTRL sub‑problem is a convex program over a known polytope) and rely on only a few hyper‑parameters (C, β) that are independent of the horizon H or the size of the state‑action space. Moreover, the analysis introduces tools (skipping estimators for occupancy measures, Tsallis regularization, transition‑uncertainty isolation) that are likely to be useful for future work on heavy‑tailed reinforcement learning with function approximation, continuous spaces, or multi‑agent interactions.

In summary, the paper provides the first BoBW algorithms for heavy‑tailed episodic MDPs, achieving (\tilde O(T^{1/α})) vs. (\tilde O(T^{1/α}+√T)) regret in adversarial regimes and O(log T) vs. O(log² T) regret in stochastic/self‑bounding regimes for known and unknown transitions respectively. The technical contributions address the core challenges of heavy‑tailed noise, long‑horizon dependence, and transition uncertainty, thereby advancing the theoretical foundations of robust reinforcement learning.