The Learning Approach to Games

This work introduces a unified framework for analyzing games in greater depth. In the existing literature, players’ strategies are typically assigned scalar values, and equilibrium concepts are used to identify compatible choices. However, this approach neglects the internal structure of players, thereby failing to accurately model observed behaviors. To address this limitation, we propose an abstract definition of a player, consistent with constructions in reinforcement learning. Instead of defining games as external settings, our framework defines them in terms of the players themselves. This offers a language that enables a deeper connection between games and learning. To illustrate the need for this generality, we study a simple two-player game and show that even in basic settings, a sophisticated player may adopt dynamic strategies that cannot be captured by simpler models or compatibility analysis. For a general definition of a player, we discuss natural conditions on its components and define competition through their behavior. In the discrete setting, we consider players whose estimates largely follow the standard framework from the literature. We explore connections to correlated equilibrium and highlight that dynamic programming naturally applies to all estimates. In the mean-field setting, we exploit symmetry to construct explicit examples of equilibria. Finally, we conclude by examining relations to reinforcement learning.

💡 Research Summary

The paper begins by criticizing the conventional treatment of games in which a player’s strategy is represented by a single scalar or a static probability distribution. While this abstraction works for many classical equilibrium analyses, it fails to capture the internal decision‑making machinery of modern reinforcement‑learning agents, which learn, adapt, and maintain internal models of the environment and opponents. To bridge this gap, the authors propose a new, player‑centric formalism that treats the player itself as the primary object of study rather than the external game setting.

A player is defined as a triple ((O, L_{\phi}, \Upsilon)) consisting of (i) an observation map (O) that produces a sequence of observations from the environment, (ii) a learning algorithm (L_{\phi}) that updates a collection of internal estimates (\phi) based on past observations, estimates, and actions, and (iii) a behavior map (\Upsilon) that selects actions given the current observations and estimates. This definition yields a natural recursive loop: observations feed the learning algorithm, which updates estimates; estimates together with past actions determine the next behavior; the behavior generates new observations, and the cycle repeats.

The authors introduce two fundamental consistency requirements. “Consistent observations” demand that the observation sequence at time (n) be a subsequence of the one at time (n+1), ensuring that information is never lost as the player ages. “Recurrent behavior” is formalized via an ((r,\delta)) condition on a metric (d) over the space of behaviors: a behavior (\Upsilon^) is ((r,\delta))-recurrent if the probability that the distance between (\Upsilon^) and the player’s actual behavior stays above (r) infinitely often is at most (\delta). The strongest case ((r=\delta=0)) corresponds to almost‑sure recurrence.

To model the internal structure of observations, the paper decomposes them into objects, connections, and relations. For each object index (j) there is a state space (E^{\text{obj}}_j); a mapping (L^{\text{obj}}_j) extracts object states from raw observations, (L^{\text{con}}) builds a graph of connections among objects, and (L^{\text{rel}}) assigns relational attributes to edges. A one‑step predictive model (L^{\text{pre}}) then maps the current collection of objects, connections, and relations (together with the chosen action) to a distribution over their next‑step counterparts. The notion of (\varepsilon)-predictiveness requires that the law of the predicted next‑step variables converges in a chosen metric to the true conditional law, up to an error (\varepsilon). This captures the idea that a sophisticated player learns a probabilistic model of how the world evolves, rather than a deterministic point estimate.

In the discrete‑game setting, the authors retain the standard set of estimates but introduce the concept of “uncertain equilibrium.” An uncertain equilibrium is a profile of behaviors that simultaneously satisfies (a) optimality – each player’s behavior maximizes expected return given its current estimates, and (b) recurrence – the behavior is visited infinitely often with probability one. This differs from Nash equilibrium, which only requires mutual best responses, because it explicitly incorporates the learning dynamics and the possibility of stochastic internal states. The authors also relate uncertain equilibrium to Aumann’s correlated equilibrium, showing that the former can be viewed as a dynamic, learning‑aware extension of the latter.

The mean‑field section treats a continuum of identical players. By exploiting symmetry, a representative player estimates the population’s aggregate strategy distribution. The paper constructs explicit uncertain equilibria in this setting and proposes a learning algorithm that updates the representative’s estimate via dynamic programming. The algorithm respects the same consistency and recurrence conditions, demonstrating that the framework scales to large populations.

Finally, the paper connects its formalism to contemporary reinforcement‑learning literature. It discusses random value‑function approaches (e.g., Thompson sampling, Bayesian deep RL) and argues that value uncertainty should be treated as part of the player’s internal estimate rather than an external parameter. A learning algorithm that does not rely on traditional value‑iteration is sketched, highlighting how multi‑agent RL can be reframed as searching for uncertain equilibria rather than static Nash points.

Overall, the contribution is a unified, mathematically precise language that places the learning agent at the heart of game‑theoretic analysis. By explicitly modeling observations, internal estimates, and behavior as interdependent stochastic processes, the framework captures dynamic strategic adaptation, provides a bridge between equilibrium concepts and reinforcement learning, and opens avenues for designing and analyzing sophisticated multi‑agent systems that go beyond the limitations of classical static game theory.