A Unified Framework for Locality in Scalable MARL

Scalable Multi-Agent Reinforcement Learning (MARL) is fundamentally challenged by the curse of dimensionality. A common solution is to exploit locality, which hinges on an Exponential Decay Property (EDP) of the value function. However, existing conditions that guarantee the EDP are often conservative, as they are based on worst-case, environment-only bounds (e.g., supremums over actions) and fail to capture the regularizing effect of the policy itself. In this work, we establish that locality can also be a \emph{policy-dependent} phenomenon. Our central contribution is a novel decomposition of the policy-induced interdependence matrix, $H^π$, which decouples the environment’s sensitivity to state ($E^{\mathrm{s}}$) and action ($E^{\mathrm{a}}$) from the policy’s sensitivity to state ($Π(π)$). This decomposition reveals that locality can be induced by a smooth policy (small $Π(π)$) even when the environment is strongly action-coupled, exposing a fundamental locality-optimality tradeoff. We use this framework to derive a general spectral condition $ρ(E^{\mathrm{s}}+E^{\mathrm{a}}Π(π)) < 1$ for exponential decay, which is strictly tighter than prior norm-based conditions. Finally, we leverage this theory to analyze a provably-sound localized block-coordinate policy improvement framework with guarantees tied directly to this spectral radius.

💡 Research Summary

The paper tackles the fundamental scalability bottleneck of multi‑agent reinforcement learning (MARL), namely the exponential growth of the joint state‑action space with the number of agents. A widely used remedy is to exploit locality: the influence of distant agents on an agent’s value function decays exponentially with graph distance, a property known as the Exponential Decay Property (EDP). Existing works certify EDP by imposing worst‑case, environment‑only coupling conditions (e.g., Dobrushin‑style bounds) that ignore the actual policy being executed. Such “action‑supremum” bounds are overly conservative because they conflate the physics of the environment with the logic of the policy.

The authors propose a fundamentally different viewpoint: locality should be regarded as a policy‑dependent phenomenon. They introduce three matrices that capture distinct sources of influence:

• (E^{\mathrm{s}}) – the environment’s sensitivity of next‑state marginals to a change in a single state coordinate (state‑to‑state coupling).
• (E^{\mathrm{a}}) – the environment’s sensitivity of next‑state marginals to a change in a single action coordinate (action‑to‑state coupling).
• (\Pi(\pi)) – the policy’s sensitivity of each local action distribution to a change in a single state coordinate (state‑to‑action coupling).

For any product‑form policy (\pi) and synchronous dynamics, the policy‑induced interdependence matrix (C^{\pi}) (which measures the one‑step influence of coordinate (i) on coordinate (j) under the closed‑loop system) satisfies the entry‑wise inequality

\