Efficiently Solving Mixed-Hierarchy Games with Quasi-Policy Approximations

Multi-robot coordination often exhibits hierarchical structure, with some robots’ decisions depending on the planned behaviors of others. While game theory provides a principled framework for such interactions, existing solvers struggle to handle mixed information structures that combine simultaneous (Nash) and hierarchical (Stackelberg) decision-making. We study N-robot forest-structured mixed-hierarchy games, in which each robot acts as a Stackelberg leader over its subtree while robots in different branches interact via Nash equilibria. We derive the Karush-Kuhn-Tucker (KKT) first-order optimality conditions for this class of games and show that they involve increasingly high-order derivatives of robots’ best-response policies as the hierarchy depth grows, rendering a direct solution intractable. To overcome this challenge, we introduce a quasi-policy approximation that removes higher-order policy derivatives and develop an inexact Newton method for efficiently solving the resulting approximated KKT systems. We prove local exponential convergence of the proposed algorithm for games with non-quadratic objectives and nonlinear constraints. The approach is implemented in a highly optimized Julia library (MixedHierarchyGames.jl) and evaluated in simulated experiments, demonstrating real-time convergence for complex mixed-hierarchy information structures.

💡 Research Summary

The paper addresses the challenging problem of solving multi‑robot coordination games that combine simultaneous (Nash) and hierarchical (Stackelberg) decision‑making. While Nash games capture mutual best‑responses and Stackelberg games model leader‑follower interactions, real‑world robotic systems often exhibit a mixture of both: some robots act as leaders for a subtree of agents, while robots belonging to different subtrees interact concurrently. Existing solvers either assume purely Nash or purely Stackelberg structures, or they restrict themselves to quadratic costs and linear constraints, which limits applicability to realistic robotic dynamics that are nonlinear and involve non‑quadratic objectives.

Problem formulation
The authors model an N‑robot mixed‑hierarchy game as a directed acyclic graph G = (V, E) where each node corresponds to a robot and each edge (i, j) denotes a direct leader‑follower relationship (i leads j). They restrict the topology to a forest: each robot has at most one direct leader, but may have multiple indirect leaders, resulting in a collection of disjoint trees. Each robot i chooses a continuous trajectory z_i ∈ ℝ^{n_i} to minimize a smooth objective J_i(z) subject to smooth equality constraints g_i(z_i)=0 (e.g., dynamics feasibility). The set of robots that are neither leaders nor followers of i, denoted N(i), interact with i via a Nash relationship, while the set of followers F(i) is governed by Stackelberg dynamics.

KKT conditions and the high‑order derivative obstacle
To characterize a mixed‑hierarchy equilibrium, the authors embed each follower’s optimal response map into its leader’s optimization problem, leading to a bilevel structure that recurses down the tree. When writing the first‑order KKT conditions for the whole game, the leader’s stationarity condition involves derivatives of the followers’ best‑response policies. Because each follower’s policy itself depends on the policies of its own followers, the chain rule generates higher‑order derivatives that grow with the depth of the hierarchy. Consequently, a direct Newton‑type solution would require computing and storing derivatives of order equal to the tree depth, which quickly becomes intractable for realistic depths.

Quasi‑policy approximation
The central methodological contribution is the introduction of a “quasi‑policy” approximation. Instead of differentiating the full nested best‑response maps, the authors retain only first‑order policy derivatives and discard all higher‑order terms. In other words, when a leader incorporates a follower’s policy Φ_j(z_{L(j)}), the Jacobian of Φ_j with respect to its leaders is kept, but the Hessian and higher‑order tensors are set to zero. This approximation mirrors recent work on feedback Nash and Stackelberg games, but extends it to arbitrary forest‑structured mixed hierarchies. The resulting approximate KKT system is much lower‑dimensional and amenable to efficient numerical treatment.

Inexact Newton algorithm
To solve the approximated KKT system, the authors design an inexact Newton method. At iteration k, given the current joint strategy z_k, they evaluate the residual F(z_k) of the approximate KKT equations and construct an approximate Jacobian J̃(z_k) that incorporates only the retained first‑order policy derivatives. The Newton step Δz solves the linear system J̃(z_k)Δz = –F(z_k). Because J̃ is not the exact Jacobian, the linear solve is performed with a Krylov subspace method (e.g., GMRES) and may be terminated early, yielding an “inexact” step. A line search with backtracking ensures sufficient decrease of a merit function. The authors prove that, under standard regularity assumptions (smoothness, uniqueness of best‑responses, and a bounded approximation error), the method converges locally with exponential (quadratic‑like) rate despite the inexactness and the policy approximation.

Theoretical guarantees
The convergence proof proceeds by showing that the quasi‑policy approximation introduces a perturbation that is Lipschitz‑continuous in a neighborhood of the true equilibrium. The inexact Newton step satisfies the Dennis–Mendelson condition, guaranteeing super‑linear convergence when the linear solve tolerance shrinks appropriately. Moreover, the authors extend the analysis to games with nonlinear equality constraints, demonstrating that the same local convergence properties hold when the constraints are regular (full‑rank Jacobian).

Software implementation
All algorithmic components are implemented in Julia and released as the open‑source library MixedHierarchyGames.jl. The library leverages ForwardDiff.jl for automatic differentiation of the smooth cost and constraint functions, and uses IterativeSolvers.jl for the Krylov linear solves. Users only need to specify each robot’s cost, constraints, and the forest graph; the library automatically constructs the quasi‑policy KKT system and runs the inexact Newton solver.

Experimental evaluation
Two representative domains are used to validate the approach:

1. Vehicle merging scenario – Four autonomous cars share a single lane. Car 1 leads the entire convoy, car 2 leads car 4, while cars 3 and the pair (2, 4) negotiate a merge simultaneously. The problem includes nonlinear vehicle dynamics, collision‑avoidance constraints, and non‑quadratic comfort costs. The mixed‑hierarchy solver converges in under 10 ms per iteration on a standard laptop, enabling real‑time replanning.

2. Target‑guarding game – Multiple defender robots protect a stationary target while an attacker robot interacts with them via a Nash game. Defenders are organized in a tree hierarchy (e.g., commander → squad leaders → individual units). The game incorporates nonlinear sensor models and energy‑consumption objectives. The solver finds an equilibrium in roughly 12 ms for a 12‑robot instance, again demonstrating real‑time capability.

Both experiments are compared against a state‑of‑the‑art Mathematical Programming Network (MPN) solver that is limited to quadratic costs and linear constraints. The proposed method achieves more than an order of magnitude speedup while maintaining solution quality; the deviation from the exact (full‑derivative) equilibrium is negligible in the simulated trajectories.

Conclusions and future work
The paper delivers a unified framework for mixed Nash‑Stackelberg games with forest‑structured information patterns, a principled quasi‑policy approximation that eliminates prohibitive high‑order derivatives, and an efficient inexact Newton algorithm with provable local exponential convergence. By releasing a high‑performance Julia library, the authors make the technique readily accessible to the robotics community. Future directions include quantifying the approximation error analytically, extending the approach to stochastic or partially observable games, and scaling to very large multi‑agent systems via parallel and distributed implementations.