Continuous Strategy Replicator Dynamics for Multi--Agent Learning

The problem of multi-agent learning and adaptation has attracted a great deal of attention in recent years. It has been suggested that the dynamics of multi agent learning can be studied using replicator equations from population biology. Most existing studies so far have been limited to discrete strategy spaces with a small number of available actions. In many cases, however, the choices available to agents are better characterized by continuous spectra. This paper suggests a generalization of the replicator framework that allows to study the adaptive dynamics of Q-learning agents with continuous strategy spaces. Instead of probability vectors, agents strategies are now characterized by probability measures over continuous variables. As a result, the ordinary differential equations for the discrete case are replaced by a system of coupled integral–differential replicator equations that describe the mutual evolution of individual agent strategies. We derive a set of functional equations describing the steady state of the replicator dynamics, examine their solutions for several two-player games, and confirm our analytical results using simulations.

💡 Research Summary

The paper addresses a fundamental gap in the study of multi‑agent learning: the majority of existing evolutionary‑game‑theoretic analyses assume a finite set of discrete actions, while many real‑world problems involve agents that can choose from a continuum of strategies. To bridge this gap, the authors extend the classic replicator dynamics—originally formulated for probability vectors over discrete strategies—to a framework that operates on probability measures defined over continuous action spaces.

The core of the contribution lies in reformulating the Q‑learning update rule in a way that yields a continuous‑strategy replicator equation. For each agent i, the strategy at time t is represented by a density μ_i(x,t) over a continuous variable x (typically confined to an interval such as