Data- and Variance-dependent Regret Bounds for Online Tabular MDPs

This work studies online episodic tabular Markov decision processes (MDPs) with known transitions and develops best-of-both-worlds algorithms that achieve refined data-dependent regret bounds in the adversarial regime and variance-dependent regret bounds in the stochastic regime. We quantify MDP complexity using a first-order quantity and several new data-dependent measures for the adversarial regime, including a second-order quantity and a path-length measure, as well as variance-based measures for the stochastic regime. To adapt to these measures, we develop algorithms based on global optimization and policy optimization, both built on optimistic follow-the-regularized-leader with log-barrier regularization. For global optimization, our algorithms achieve first-order, second-order, and path-length regret bounds in the adversarial regime, and in the stochastic regime, they achieve a variance-aware gap-independent bound and a variance-aware gap-dependent bound that is polylogarithmic in the number of episodes. For policy optimization, our algorithms achieve the same data- and variance-dependent adaptivity, up to a factor of the episode horizon, by exploiting a new optimistic $Q$-function estimator. Finally, we establish regret lower bounds in terms of data-dependent complexity measures for the adversarial regime and a variance measure for the stochastic regime, implying that the regret upper bounds achieved by the global-optimization approach are nearly optimal.

💡 Research Summary

This paper studies online episodic tabular Markov decision processes (MDPs) with known transition dynamics and seeks a single algorithm that simultaneously attains refined, data‑dependent regret bounds in the adversarial setting and variance‑dependent bounds in the stochastic setting. The authors introduce several new complexity measures: the classic first‑order “small‑loss” quantity L* (cumulative loss of the best fixed policy), a second‑order quantity Q∞ that captures loss fluctuation magnitude, a path‑length measure V₁ that quantifies how much the loss sequence changes over time, and, for the stochastic regime, occupancy‑weighted variance V and conditional occupancy‑weighted variance V_c.

Two families of algorithms are proposed. The first family is based on global optimization over the space of occupancy measures. Both families employ an optimistic follow‑the‑regularized‑leader (OFTRL) framework with a log‑barrier regularizer and an adaptive learning rate. For global optimization, the algorithm predicts future losses either via a gradient‑descent style update (useful for path‑length bounds) or via an empirical mean estimator (useful for variance‑aware bounds). The resulting regret guarantees are:

• In the adversarial regime,
  (\tilde O\big(p S A \min{L^, H T - L^, Q_\infty, V_1}\big)),
  where p denotes the log‑barrier parameter.
• In the stochastic regime, a variance‑aware gap‑independent bound
  (\tilde O\big(\sqrt{S A V_T}\big)) and a variance‑aware gap‑dependent bound that scales only polylogarithmically with T.

The second family uses policy optimization, updating per‑state action distributions directly. To correct the bias introduced by optimistic loss predictions, the authors design a more optimistic Q‑function estimator than prior work. This yields analogous regret bounds up to an additional factor of (H^2):

• Adversarial regret (\tilde O\big(p H^2 S A \min{L^, H T - L^, Q_\infty, V_1}\big)).
• Stochastic gap‑independent bound (\tilde O\big(\sqrt{H^2 S A V_T}\big)) and a polylog‑T gap‑dependent bound.

Finally, the paper establishes lower bounds of (\Omega(\sqrt{S A L^*})), (\Omega(\sqrt{S A Q_\infty})), (\Omega(\sqrt{H V_1})) for the adversarial case and (\Omega(\sqrt{S A V_T})) for the stochastic case, showing that the global‑optimization algorithm’s upper bounds are nearly optimal. Tables compare these results with prior work, highlighting that the proposed methods uniquely achieve first‑order, second‑order, path‑length, and variance‑aware adaptivity within a single best‑of‑both‑worlds framework. The contributions advance the theory of online MDPs by tightly integrating data‑dependent and variance‑dependent analyses, offering practically relevant algorithms that automatically adapt to the underlying loss structure without prior knowledge of the environment.