Adversarial Latent-State Training for Robust Policies in Partially Observable Domains

Robustness under latent distribution shift remains challenging in partially observable reinforcement learning. We formalize a focused setting where an adversary selects a hidden initial latent distribution before the episode, termed an adversarial latent-initial-state POMDP. Theoretically, we prove a latent minimax principle, characterize worst-case defender distributions, and derive approximate best-response inequalities with finite-sample concentration bounds that make the optimization and sampling terms explicit. Empirically, using a Battleship benchmark, we demonstrate that targeted exposure to shifted latent distributions reduces average robustness gaps between Spread and Uniform distributions from 10.3 to 3.1 shots at equal budget. Furthermore, iterative best-response training exhibits budget-sensitive behavior that is qualitatively consistent with the theorem-guided diagnostics once one accounts for discounted PPO surrogates and finite-sample noise. Ultimately, we show that for latent-initial-state problems, the framework yields a clean evaluation game and useful theorem-motivated diagnostics while also making clear where implementation-level surrogates and optimization limits enter.

💡 Research Summary

The paper introduces a novel class of partially observable reinforcement learning problems called adversarial latent‑initial‑state POMDPs. In this setting, a hidden latent variable z (e.g., the placement of ships in Battleship) is drawn once at the start of an episode from a distribution ρ chosen by a defender. The transition and observation dynamics thereafter are deterministic conditioned on z, and the attacker (policy) seeks to minimize expected episode length τ (the number of shots needed to sink all ships). This formulation isolates the challenge of distribution shift that originates from a fixed latent condition rather than from stochastic transitions or rewards.

The authors formalize the interaction as a finite zero‑sum game between the attacker’s deterministic history‑dependent policies Π_det and the defender’s convex set P of admissible latent distributions. Under finiteness of horizon, action, observation, and latent spaces, they prove a latent minimax principle (Theorem 1):
\