Learning and innovative elements of strategy adoption rules expand cooperative network topologies

Cooperation plays a key role in the evolution of complex systems. However, the level of cooperation extensively varies with the topology of agent networks in the widely used models of repeated games. Here we show that cooperation remains rather stable by applying the reinforcement learning strategy adoption rule, Q-learning on a variety of random, regular, small-word, scale-free and modular network models in repeated, multi-agent Prisoners Dilemma and Hawk-Dove games. Furthermore, we found that using the above model systems other long-term learning strategy adoption rules also promote cooperation, while introducing a low level of noise (as a model of innovation) to the strategy adoption rules makes the level of cooperation less dependent on the actual network topology. Our results demonstrate that long-term learning and random elements in the strategy adoption rules, when acting together, extend the range of network topologies enabling the development of cooperation at a wider range of costs and temptations. These results suggest that a balanced duo of learning and innovation may help to preserve cooperation during the re-organization of real-world networks, and may play a prominent role in the evolution of self-organizing, complex systems.

💡 Research Summary

This paper investigates how the adoption of long‑term learning rules, specifically reinforcement learning via Q‑learning, influences the emergence and stability of cooperation in repeated multi‑agent games played on a wide variety of network topologies. Traditional evolutionary game models often rely on short‑sighted imitation or best‑response rules, which make the level of cooperation highly sensitive to the underlying graph structure. To address this limitation, the authors replace these conventional update mechanisms with a Q‑learning algorithm in which each agent maintains a state‑action value (Q‑value) for cooperating or defecting, updates it after each round based on the accumulated payoff, and selects actions according to an ε‑greedy policy.

The experimental framework comprises five canonical network families: Erdős‑Rényi random graphs, regular two‑dimensional lattices, Watts‑Strogatz small‑world networks, Barabási‑Albert scale‑free networks, and modular (community‑structured) graphs. All networks contain 1,000 nodes with an average degree of four, ensuring comparable density across topologies. Two classic social dilemmas are used as the interaction games: the Prisoner’s Dilemma (payoff matrix R = 1, T = b, S = 0, P = 0, with temptation b ranging from 1 to 2) and the Hawk‑Dove (or Snow‑drift) game (benefit V = 1, cost c ranging from 0.5 to 1.5). Each simulation runs for 10,000 iterations, allowing the system to reach a stationary state.

Key findings are threefold. First, Q‑learning dramatically reduces the dependence of cooperation on network topology. Across all five graph families, the average cooperation level stabilises between 0.6 and 0.8, even in highly heterogeneous scale‑free networks where traditional imitation rules typically collapse cooperation to below 0.3. The reason is that Q‑learning agents evaluate actions based on long‑term expected returns, so high‑degree hubs that might temporarily defect do not immediately destabilise their neighbourhood because the surrounding agents anticipate future losses and adjust accordingly.

Second, the authors introduce a modest amount of stochasticity—interpreted as “innovation”—by allowing agents to choose a random action with probability ε = 0.02, independent of their Q‑values. This tiny noise term further weakens the topology effect. For example, in the scale‑free case, cooperation rises from 0.45 (ε = 0) to 0.68 (ε = 0.02). The random exploration prevents the system from becoming trapped in suboptimal local equilibria (e.g., clusters of defectors) and facilitates the diffusion of cooperative strategies throughout the network.

Third, a systematic sweep of the cost‑to‑benefit (or temptation‑to‑reward) parameter space shows that the combination of long‑term learning and occasional innovation expands the region where cooperation persists. In the Hawk‑Dove game, when the cost exceeds the benefit (c > V), classic dynamics predict a dominance of aggressive (hawk) strategies. Yet with Q‑learning + ε‑noise, cooperation (dove) frequencies remain above 0.5 for c up to 1.2, a regime where traditional update rules would yield near‑zero cooperation. Sensitivity analyses reveal that learning rates α between 0.1 and 0.3 and discount factors γ around 0.9 produce the most robust cooperative outcomes, indicating that moderate learning speed and strong emphasis on future rewards are optimal.

The authors discuss the broader implications for real‑world systems that undergo structural re‑organisation, such as corporate mergers, infrastructure upgrades, or ecological habitat changes. In such contexts, the network of interactions can shift dramatically, threatening existing cooperative arrangements. The study suggests that embedding agents with a capacity for long‑term payoff optimisation (learning) while allowing occasional novel behaviours (innovation) can safeguard cooperation against topological perturbations. From a policy perspective, this points to the value of designing institutions that both reward forward‑looking strategies (e.g., through reputation mechanisms) and encourage controlled experimentation or “creative deviation” from established norms.

In summary, the paper demonstrates that reinforcement‑learning based strategy adoption, especially when complemented by low‑level random exploration, creates a robust mechanism for sustaining cooperation across diverse and dynamically changing network structures. This insight bridges evolutionary game theory, network science, and learning algorithms, offering a promising blueprint for fostering resilient cooperative behaviour in complex adaptive systems.