Playing Markov Games Without Observing Payoffs

Optimization under uncertainty is a fundamental problem in learning and decision-making, particularly in multi-agent systems. Previously, Feldman, Kalai, and Tennenholtz [2010] demonstrated the ability to efficiently compete in repeated symmetric two-player matrix games without observing payoffs, as long as the opponents actions are observed. In this paper, we introduce and formalize a new class of zero-sum symmetric Markov games, which extends the notion of symmetry from matrix games to the Markovian setting. We show that even without observing payoffs, a player who knows the transition dynamics and observes only the opponents sequence of actions can still compete against an adversary who may have complete knowledge of the game. We formalize three distinct notions of symmetry in this setting and show that, under these conditions, the learning problem can be reduced to an instance of online learning, enabling the player to asymptotically match the return of the opponent despite lacking payoff observations. Our algorithms apply to both matrix and Markov games, and run in polynomial time with respect to the size of the game and the number of episodes. Our work broadens the class of games in which robust learning is possible under severe informational disadvantage and deepens the connection between online learning and adversarial game theory.

💡 Research Summary

The paper tackles the challenging problem of learning to play zero‑sum symmetric Markov games when the learner receives no payoff feedback and knows only the opponent’s actions. Building on the classic “Copycat” algorithm of Feldman, Kalai, and Tennenholtz (2010), which achieves sublinear regret in symmetric matrix games without observing payoffs, the authors extend the idea to the Markovian setting. They first formalize three increasingly general notions of symmetry for Markov games: (1) Per‑state Symmetric Games (SSG), where each state individually defines a symmetric zero‑sum matrix game; (2) Symmetry with respect to Markov policies (MSG), requiring that for any pair of Markov policies the value functions are exact negatives of each other; and (3) Symmetry with respect to history‑dependent policies (HSG), demanding the same negative relationship for any pair of possibly history‑dependent strategies.

For SSG, the authors observe that the global game can be decomposed into independent state‑wise matrix games. By running the original Copycat algorithm separately in each state, they obtain a regret bound of order O(n p T |S| H), where n is the number of actions, |S| the number of states, H the horizon, T the number of episodes, and p a polylogarithmic factor. The algorithm runs in polynomial time in |S|, n and T.

The MSG setting appears more complex because the value depends on the interaction of immediate rewards and future dynamics. Surprisingly, the paper proves a structural reduction: any MSG can be transformed into an equivalent SSG by recursively “peeling back” the horizon and exposing skew‑symmetric payoff matrices at each layer. This transformation preserves the game’s value for any pair of strategies, so the same Copycat‑based algorithm for SSG applies unchanged, yielding the identical regret bound O(n p T |S| H).

The HSG class imposes the strongest symmetry condition. The authors show that under HSG the expected future value from any step beyond the first becomes a constant independent of actions, effectively collapsing the entire episodic Markov game into a single symmetric matrix game. Consequently, a single application of the standard Copycat algorithm suffices, achieving a regret of O(H n √T). This bound depends only on the horizon, the action set size, and the square root of the number of episodes.

To complement the upper bounds, the paper provides a matching lower bound for SSG: there exists an adversarial payoff structure that forces any algorithm to incur regret Ω(n p T |S| H). Hence the proposed algorithms are optimal with respect to the relevant parameters.

Overall, the work reveals a counter‑intuitive hierarchy: stronger symmetry requirements (MSG, HSG) actually simplify the learning problem rather than broaden it. By exploiting these structural constraints, the authors successfully generalize the Copycat scheme to Markov games, establishing polynomial‑time, sublinear‑regret learning without any payoff observations. The results deepen the connection between online learning and adversarial game theory and open new avenues for robust multi‑agent learning under severe informational disadvantages.