Multi-Agent Guided Policy Search for Non-Cooperative Dynamic Games

Multi-agent reinforcement learning (MARL) optimizes strategic interactions in non-cooperative dynamic games, where agents have misaligned objectives. However, data-driven methods such as multi-agent policy gradients (MA-PG) often suffer from instability and limit-cycle behaviors. Prior stabilization techniques typically rely on entropy-based exploration, which slows learning and increases variance. We propose a model-based approach that incorporates approximate priors into the reward function as regularization. In linear quadratic (LQ) games, we prove that such priors stabilize policy gradients and guarantee local exponential convergence to an approximate Nash equilibrium. We then extend this idea to infinite-horizon nonlinear games by introducing Multi-agent Guided Policy Search (MA-GPS), which constructs short-horizon local LQ approximations from trajectories of current policies to guide training. Experiments on nonlinear vehicle platooning and a six-player strategic basketball formation show that MA-GPS achieves faster convergence and more stable learning than existing MARL methods.

💡 Research Summary

The paper tackles a fundamental difficulty in multi‑agent reinforcement learning (MARL) for non‑cooperative dynamic games: policy‑gradient methods (MA‑PG) often exhibit unstable dynamics or converge to limit cycles because agents simultaneously update misaligned policies. Existing stabilization techniques rely on entropy‑based exploration, which slows learning and inflates gradient variance. The authors propose a model‑based regularization that injects an approximate prior—derived from a stabilizing feedback policy—directly into each agent’s reward function.

Linear‑Quadratic (LQ) Games Theory
In the infinite‑horizon LQ setting, the dynamics are linear and each agent’s stage cost is quadratic. The authors define an arbitrary stabilizing feedback matrix (\breve K) as a “guide”. For each agent (i) they augment the cost with (\rho|K_i-\breve K_i|_{R_i}^2), where (\rho>0) is a regularization weight and (R_i) is the control‑cost matrix. This term adds (\rho R) to the Jacobian of the pseudo‑gradient (w(K)). The key theoretical result (Proposition 1) shows that: (1) (w(K)) is locally Lipschitz on the set of stabilizing policies; (2) when (\rho) is sufficiently large, all eigenvalues of (\nabla w(K)+\rho R) have positive real parts, guaranteeing local exponential convergence of the gradient dynamics; (3) the bias between the regularized equilibrium (\hat K^) and the true Nash equilibrium (K^) can be bounded in closed form and depends on (\rho) and the distance between (\breve K) and (K^*). The analysis reveals a clear trade‑off: a small (\rho) may fail to stabilize the dynamics, while a very large (\rho) forces the learned policy toward the guide, increasing bias. Empirical phase‑plane plots confirm that modest (\rho) values already eliminate limit cycles and that the converged policies remain close to the true Nash equilibrium even when the guide is imperfect.

Extension to Nonlinear Games – MA‑GPS
For general nonlinear games, exact global LQ approximations are infeasible. Inspired by iLQGames, the authors construct a short‑horizon local LQ model around the trajectory generated by the current neural‑network policies. They then solve the coupled Riccati equations of this local game to obtain a time‑varying feedback guide (\hat K_t). The regularized reward uses the same quadratic deviation term, now evaluated against the local guide at each timestep. The resulting algorithm, Multi‑agent Guided Policy Search (MA‑GPS), proceeds iteratively: (1) roll out current policies to collect state‑action trajectories; (2) linearize dynamics and quadratize costs along the trajectory; (3) compute the local Nash feedback (\hat K_t); (4) perform MA‑PG (or actor‑critic) updates with the regularized reward; (5) update policy parameters and repeat. This scheme requires no pre‑computed guide and can be applied online, preserving the scalability of model‑free MARL while inheriting the stability benefits of model‑based guidance.

Experimental Validation
The authors evaluate three domains: (i) pure LQ games, (ii) a nonlinear vehicle platooning scenario with five autonomous cars, and (iii) a six‑player basketball formation task where agents must coordinate offensive and defensive positions. In LQ games, MA‑GPS converges 2–3× faster than unregularized MA‑PG and eliminates oscillatory behavior. In the platooning task, MA‑GPS achieves lower inter‑vehicle spacing error and fewer collisions, outperforming MADDPG, COMA, and standard MA‑PG by 12–18 % in cumulative cost. In the basketball domain, the learned policies yield a higher average point differential (≈1.5 points per episode) and smoother learning curves compared with baselines. Notably, even when the local guide is derived from a slightly inaccurate dynamics model, MA‑GPS still surpasses the guide’s performance, confirming that the method can correct imperfect priors.

Discussion and Limitations
The approach hinges on the quality of the guide and the choice of (\rho). While the theory guarantees local stability, global convergence to the true Nash equilibrium is not proven, especially in games with multiple equilibria or non‑unique solutions. The need to linearize around current trajectories may be problematic if the initial policies are very poor, leading to misleading local models. Moreover, the bias‑stability trade‑off requires empirical tuning of (\rho). Future work could explore adaptive schemes for (\rho), automatic selection of guide policies, extensions to stochastic dynamics, and real‑world robotic deployments.

Conclusion
By embedding a model‑based prior directly into the reward function, the paper provides a principled way to stabilize multi‑agent policy‑gradient learning in non‑cooperative dynamic games. The theoretical analysis for LQ games establishes exponential convergence under a simple regularization, while the MA‑GPS algorithm extends these benefits to complex nonlinear settings through efficient local LQ approximations. Empirical results across linear, vehicular, and strategic sports domains demonstrate faster, more reliable learning and higher‑quality policies than state‑of‑the‑art MARL methods, marking a significant step toward practical, stable multi‑agent decision making in competitive environments.