Payoff-based Inhomogeneous Partially Irrational Play for Potential Game Theoretic Cooperative Control of Multi-agent Systems

This paper handles a kind of strategic game called potential games and develops a novel learning algorithm Payoff-based Inhomogeneous Partially Irrational Play (PIPIP). The present algorithm is based on Distributed Inhomogeneous Synchronous Learning (DISL) presented in an existing work but, unlike DISL,PIPIP allows agents to make irrational decisions with a specified probability, i.e. agents can choose an action with a low utility from the past actions stored in the memory. Due to the irrational decisions, we can prove convergence in probability of collective actions to potential function maximizers. Finally, we demonstrate the effectiveness of the present algorithm through experiments on a sensor coverage problem. It is revealed through the demonstration that the present learning algorithm successfully leads agents to around potential function maximizers even in the presence of undesirable Nash equilibria. We also see through the experiment with a moving density function that PIPIP has adaptability to environmental changes.

💡 Research Summary

**
The paper addresses cooperative control of multi‑agent systems by exploiting the structure of potential games, where a global objective can be expressed as a potential function whose maximizers correspond to desirable joint actions. While potential games guarantee the existence of pure Nash equilibria, not all equilibria are optimal; some may be “undesirable” local optima that trap conventional learning dynamics. To overcome this limitation, the authors propose a novel payoff‑based learning algorithm called Payoff‑based Inhomogeneous Partially Irrational Play (PIPIP).

PIPIP builds on the Distributed Inhomogeneous Synchronous Learning (DISL) framework but introduces a controlled amount of irrationality. Each agent stores its two most recent actions and the associated payoffs. In the normal case the agent selects the action with the higher observed payoff, but with a time‑varying probability ε(t) it deliberately chooses the lower‑payoff past action. The sequence ε(t) is designed to decay over time, providing extensive exploration early on and gradually reducing randomness as the system converges. This “partial irrationality” creates a non‑zero probability of escaping from sub‑optimal Nash equilibria.

The authors formalize the learning process as a finite‑state Markov chain. The baseline DISL dynamics constitute an unperturbed chain {P₀^t}. PIPIP is a regular perturbation of this chain: with probability 1‑ε(t) the transition follows DISL, and with probability ε(t) it follows an “irrational” transition. Using the theory of regular perturbations, resistance trees, and stochastic potential, they prove that the set of stochastically stable states of the perturbed chain coincides with the set of optimal Nash equilibria (i.e., the maximizers of the potential function). Consequently, as ε(t)→0 the probability that the joint action converges to an optimal equilibrium tends to one.

Key technical assumptions include: (1) each agent’s feasible action set R_i(a_i) is reversible, connected, and contains at least three actions; (2) utility differences caused by a unilateral deviation are bounded by one (ensuring appropriate scaling). Under these conditions the potential game property guarantees that any unilateral utility change equals the corresponding change in the global potential, which is essential for the resistance analysis.

From an implementation standpoint, PIPIP requires only finite memory (two past actions and payoffs) and uses actual payoffs exclusively; no virtual payoffs or observations of other agents’ actions are needed. The algorithm operates synchronously, making it suitable for real‑time robotic platforms. Moreover, the action‑constraint handling is intrinsic: agents select only from the set R_i(a_i) dictated by physical or safety constraints.

The experimental validation focuses on a sensor coverage problem. The mission space contains a density function that agents aim to cover, and obstacles create local optima where many sensors cluster, representing undesirable Nash equilibria. In the static‑density scenario, DISL becomes trapped in such configurations, whereas PIPIP’s occasional irrational moves enable the swarm to disperse and achieve higher overall coverage, confirming convergence to a potential‑maximizing configuration. In a dynamic scenario where the density function moves over time, PIPIP continuously adapts: despite the decaying ε(t), the residual exploration allows the agents to track the moving high‑density region without any prior knowledge of its motion.

The paper also compares PIPIP with related algorithms: (i) Restricted Spatial Adaptive Play (RSAP) guarantees convergence to a distribution that can be tuned to favor optimal equilibria but requires virtual payoffs and asynchronous updates, limiting applicability; (ii) Payoff‑based Log‑Linear Learning (PLLL) also incorporates irrational moves but does not explicitly address action constraints and lacks a rigorous proof of convergence to optimal equilibria. PIPIP uniquely combines finite memory, payoff‑only information, synchronous updates, explicit constraint handling, and a provable convergence to optimal Nash equilibria.

In summary, the contributions are twofold: (1) the design of PIPIP, a lightweight, payoff‑based learning rule that injects controlled irrationality to escape sub‑optimal equilibria, and (2) a rigorous stochastic‑stability analysis showing that the algorithm converges in probability to the set of potential‑function maximizers. The experimental results demonstrate both rapid convergence in static environments and robust adaptability in time‑varying settings, highlighting the practical relevance of PIPIP for cooperative control tasks where prior knowledge of the environment is limited or the environment changes dynamically. This work thus advances the applicability of potential‑game‑based control from theoretical constructs to deployable multi‑robot systems.