Reinforcement Learning and Nonparametric Detection of Game-Theoretic Equilibrium Play in Social Networks

This paper studies two important signal processing aspects of equilibrium behavior in non-cooperative games arising in social networks, namely, reinforcement learning and detection of equilibrium play. The first part of the paper presents a reinforcement learning (adaptive filtering) algorithm that facilitates learning an equilibrium by resorting to diffusion cooperation strategies in a social network. Agents form homophilic social groups, within which they exchange past experiences over an undirected graph. It is shown that, if all agents follow the proposed algorithm, their global behavior is attracted to the correlated equilibria set of the game. The second part of the paper provides a test to detect if the actions of agents are consistent with play from the equilibrium of a concave potential game. The theory of revealed preference from microeconomics is used to construct a non-parametric decision test and statistical test which only require the probe and associated actions of agents. A stochastic gradient algorithm is given to optimize the probe in real time to minimize the Type-II error probabilities of the detection test subject to specified Type-I error probability. We provide a real-world example using the energy market, and a numerical example to detect malicious agents in an online social network.

💡 Research Summary

This paper tackles two fundamental signal‑processing problems that arise when agents in a social network engage in a non‑cooperative game: (i) learning to play an equilibrium in a distributed fashion, and (ii) detecting from data whether the observed actions are consistent with equilibrium play.

Part I – Distributed Reinforcement Learning via Diffusion Cooperation
The authors consider a symmetric normal‑form game with K agents, identical action sets, and unknown utility functions. Agents are embedded in an undirected graph G = (K,E) that captures who can exchange information. Homophily is modeled by allowing agents to form social groups that share past experiences (regret values). Building on the classic regret‑matching algorithm, the paper proposes a diffusion‑based stochastic‑approximation scheme: each agent updates a regret vector using its own payoff, then averages this vector with those received from its neighbors, and finally adjusts its mixed strategy according to a regret‑matching rule. The key theoretical result is that if every agent follows this protocol, the joint empirical distribution of actions converges, with high probability, to within an ε‑ball of the correlated‑equilibrium polytope. The analysis leverages stochastic‑approximation theory to handle the time‑varying network averaging and diminishing step‑sizes. Compared with prior work that assumes either a fully connected network or isolated agents, the diffusion approach works under sparse, possibly time‑varying topologies and exploits homophilic clustering to accelerate learning. Moreover, the algorithm respects the ordinal nature of human decisions: only the ranking of regrets matters, not their absolute magnitudes.

Part II – Non‑Parametric Detection of Equilibrium Play
The second part asks whether a sequence of observed external probes pₜ∈ℝᵐ and corresponding agent actions xₜ∈ℝᵐ can be rationalized as equilibrium outcomes of a concave potential game. The authors adopt revealed‑preference theory from micro‑economics, specifically Afriat’s theorem and its extensions, to construct a finite‑sample, non‑parametric test. If the data are generated by some concave potential function u(·), then a set of linear inequalities (Afriat’s inequalities) must be satisfied. The paper first presents a deterministic decision test (exact satisfaction) and then a statistical test that tolerates measurement noise while guaranteeing a pre‑specified Type‑I error probability α. To reduce the Type‑II error β, the authors propose a real‑time probe‑design algorithm based on Simultaneous Perturbation Stochastic Approximation (SPSA). The SPSA iteratively perturbs the probe vector, observes the resulting actions, and updates the probe in the direction that most improves the likelihood of satisfying Afriat’s inequalities. Convergence analysis shows that, for a sufficiently long observation horizon, β can be driven arbitrarily low while keeping α fixed.

Illustrative Applications

1. Energy market: Using hourly electricity price signals as probes and consumer demand as actions, the authors demonstrate that the observed data satisfy the Afriat test, indicating that consumers behave as if playing a concave potential game.
2. Online social network: Synthetic logs of user activity are generated with a subset of malicious agents that deviate from equilibrium behavior. The proposed statistical test successfully flags the malicious agents, and the SPSA‑optimized probes improve detection accuracy by more than 15 % compared with static probes.

Simulation of Learning Dynamics
A Monte‑Carlo experiment with 50 agents partitioned into five homophilic groups shows that the diffusion regret‑matching algorithm drives average regret below 0.01 within a few hundred iterations, and the empirical joint distribution settles within 0.05 of the correlated‑equilibrium set.

Conclusions and Future Directions
The paper unifies distributed equilibrium learning and equilibrium detection under a common stochastic‑approximation framework. It demonstrates that diffusion cooperation enables scalable learning to correlated equilibria even on sparse graphs, while revealed‑preference methods provide a powerful, model‑free tool for verifying equilibrium behavior from data. Future work could extend the methodology to asymmetric or dynamic games, incorporate time‑varying network topologies, and explore privacy‑preserving information exchange mechanisms.