Simultaneous AlphaZero: Extending Tree Search to Markov Games

Simultaneous AlphaZero extends the AlphaZero framework to multistep, two-player zero-sum deterministic Markov games with simultaneous actions. At each decision point, joint action selection is resolved via matrix games whose payoffs incorporate both immediate rewards and future value estimates. To handle uncertainty arising from bandit feedback during Monte Carlo Tree Search (MCTS), Simultaneous AlphaZero incorporates a regret-optimal solver for matrix games with bandit feedback. Simultaneous AlphaZero demonstrates robust strategies in a continuous-state discrete-action pursuit-evasion game and satellite custody maintenance scenarios, even when evaluated against maximally exploitative opponents.

💡 Research Summary

Simultaneous AlphaZero (SAZ) extends the celebrated AlphaZero algorithm to two‑player zero‑sum deterministic Markov games where both agents choose actions at the same time. The paper’s central insight is to treat every decision point as a matrix game: the payoff matrix combines the immediate reward with a discounted estimate of the value of the resulting successor state. This matrix is constructed on‑the‑fly during Monte‑Carlo Tree Search (MCTS) using the current neural‑network value predictions. Because the true payoffs are unknown, SAZ solves each matrix game with bandit feedback using a regret‑optimal algorithm (the UCB‑augmented method of O’Donoghue et al.). The algorithm maintains separate policy heads for each player, multiplies their independent priors to obtain a joint prior, and updates joint action counts N(s,a₁,a₂) to compute an exploration‑bonus term. Regret matching (or a few iterations thereof) quickly yields an approximate Nash equilibrium for the matrix game, which is far cheaper than solving a linear program at every tree node.

The authors provide a rigorous analysis of how approximation errors in the value function propagate up the search tree. Lemma 1 shows that the error at depth d is bounded by γ times the infinity‑norm of the error at depth d + 1, and Theorem 1 aggregates this to a root‑level bound of γ·D·‖E_D‖∞ for a tree of depth D. This demonstrates that deeper MCTS reduces value‑estimation error geometrically, even when the value prior is imperfect.

The neural architecture consists of a shared trunk (two fully‑connected layers) feeding three heads: policy π₁ for player 1, policy π₂ for player 2, and a value head that predicts a discrete distribution over return bins (Gaussian histogram loss). Because the game is zero‑sum, the value for player 2 is simply the negative of player 1’s value.

Empirical evaluation is performed on two benchmarks: (1) a continuous‑state, discrete‑action pursuit‑evasion game, and (2) a satellite custody‑maintenance scenario where one satellite tries to evade while another maintains custody. Both environments feature continuous state spaces that make traditional tabular methods infeasible. SAZ learns strong self‑play policies, achieving high win rates and, crucially, very low exploitability when pitted against a best‑response opponent (exploitability ≈ 0.01–0.03). This confirms that the combination of bandit‑feedback matrix‑game solving and the error‑propagation guarantees yields robust strategies even under imperfect value estimates.

In summary, the paper makes three key contributions: (1) a general framework for applying AlphaZero‑style search to simultaneous‑move zero‑sum Markov games, (2) the integration of a regret‑optimal bandit matrix‑game solver that scales to large action spaces, and (3) a theoretical bound on how value‑function approximation errors affect the resulting policy’s exploitability. The approach opens the door to scalable game‑theoretic planning in domains such as aerial combat, space domain awareness, and multi‑robot coordination, and suggests promising extensions to multi‑player (>2) settings, stochastic transitions, and mixed cooperative‑competitive games.