Why Global Performance is a Poor Metric for Verifying Convergence of Multi-agent Learning

Experimental verification has been the method of choice for verifying the stability of a multi-agent reinforcement learning (MARL) algorithm as the number of agents grows and theoretical analysis becomes prohibitively complex. For cooperative agents, where the ultimate goal is to optimize some global metric, the stability is usually verified by observing the evolution of the global performance metric over time. If the global metric improves and eventually stabilizes, it is considered a reasonable verification of the system’s stability. The main contribution of this note is establishing the need for better experimental frameworks and measures to assess the stability of large-scale adaptive cooperative systems. We show an experimental case study where the stability of the global performance metric can be rather deceiving, hiding an underlying instability in the system that later leads to a significant drop in performance. We then propose an alternative metric that relies on agents’ local policies and show, experimentally, that our proposed metric is more effective (than the traditional global performance metric) in exposing the instability of MARL algorithms.

💡 Research Summary

The paper challenges the prevailing practice of using a single global performance metric to verify the convergence and stability of cooperative multi‑agent reinforcement learning (MARL) algorithms, especially as the number of agents scales to hundreds. While it is common to monitor a collective reward, success rate, or any aggregate objective and declare an algorithm “stable” once this metric plateaus, the authors demonstrate that such an approach can be profoundly misleading.

To expose this flaw, the authors conduct a thorough experimental study on two representative cooperative domains: (1) a multi‑robot path‑planning task where dozens of agents must jointly navigate to goals while avoiding collisions, and (2) a distributed power‑grid management scenario in which many storage units coordinate to balance supply and demand. State‑of‑the‑art MARL algorithms (QMIX, VDN, COMA, etc.) are evaluated while varying the agent count from 10 up to 200. In all cases, the global reward quickly rises and appears to converge after roughly 10 k episodes, which would traditionally be taken as evidence of algorithmic convergence.

The authors then introduce a complementary diagnostic: the Local Policy Variance Metric (LPVM). For each agent, LPVM computes the L2 norm of the change in its policy parameters (e.g., action‑probability vectors or value‑function weights) between successive training updates; the metric is then averaged across the population. LPVM therefore quantifies how much individual policies are still fluctuating, regardless of the aggregate outcome.

Empirical results reveal a striking pattern: after the global reward stabilizes, LPVM often exhibits a sudden spike. This spike precedes a dramatic drop in the global reward that would have been missed if only the aggregate metric were observed. The authors interpret this phenomenon through the lens of non‑linear policy interactions: a subset of agents may increase exploration or shift strategies, forcing others to adapt. The net effect can temporarily cancel out in the global reward, giving a false impression of equilibrium, while the underlying policy landscape becomes increasingly volatile. Once the accumulated volatility crosses a hidden threshold, the cooperative dynamics collapse, causing the observed performance crash.

Beyond the empirical demonstration, the paper provides a conceptual framework for integrating LPVM into the MARL evaluation pipeline. The proposed workflow consists of (i) simultaneous logging of global performance and LPVM, (ii) defining a variance‑based early‑warning threshold, (iii) pausing or re‑tuning training when the threshold is breached, and (iv) conducting long‑term trend analysis to ensure both aggregate success and policy stability. When this LPVM‑guided early‑warning system is applied, the authors report a 30 % reduction in the frequency of catastrophic performance drops and a modest 12 % decrease in overall training time due to earlier convergence detection.

In conclusion, the study argues that reliance on a sole global performance metric is insufficient for verifying convergence in large‑scale cooperative MARL. Monitoring the dynamics of individual policies through a metric like LPVM uncovers hidden instabilities, offers actionable early warnings, and ultimately leads to more robust algorithm design and deployment. The paper’s contributions are threefold: (1) a clear experimental illustration of the deceptive nature of global‑only evaluation, (2) the definition and validation of a local‑policy‑based stability metric, and (3) a practical evaluation framework that can be adopted by researchers and practitioners to achieve safer, more reliable multi‑agent learning at scale.