Strategizing against No-regret Learners

How should a player who repeatedly plays a game against a no-regret learner strategize to maximize his utility? We study this question and show that under some mild assumptions, the player can always guarantee himself a utility of at least what he would get in a Stackelberg equilibrium of the game. When the no-regret learner has only two actions, we show that the player cannot get any higher utility than the Stackelberg equilibrium utility. But when the no-regret learner has more than two actions and plays a mean-based no-regret strategy, we show that the player can get strictly higher than the Stackelberg equilibrium utility. We provide a characterization of the optimal game-play for the player against a mean-based no-regret learner as a solution to a control problem. When the no-regret learner’s strategy also guarantees him a no-swap regret, we show that the player cannot get anything higher than a Stackelberg equilibrium utility.

💡 Research Summary

The paper investigates the optimal strategy for a player (the optimizer) who repeatedly plays a finite‑action bimatrix game against an opponent employing a no‑regret learning algorithm (the learner). The central benchmark is the Stackelberg equilibrium: the optimizer commits to a mixed strategy, the learner best‑responds, and the optimizer’s payoff in this equilibrium is denoted V.

The first main result (Theorem 4) shows that, regardless of which external‑regret algorithm the learner uses, the optimizer can guarantee an average per‑round payoff arbitrarily close to V. The proof constructs a mixed strategy that makes the learner’s Stackelberg best response the unique optimal action; the no‑regret guarantee then forces the learner to play this action almost every round, yielding total utility at least (V − ε)·T − o(T) for any ε>0.

In constant‑sum games, Theorem 5 proves that V is also an upper bound: the learner’s cumulative utility must be at least C·T − o(T) (where C is the constant sum), which forces the optimizer’s utility to be at most V·T + o(T).

When the learner satisfies the stronger no‑swap‑regret condition, Theorem 7 demonstrates that the optimizer cannot exceed V either. The argument relates swap‑regret to the distance between the empirical joint distribution of play and the set of distributions that make the learner indifferent, using a convex‑analysis lemma to bound the optimizer’s excess payoff by a constant times the swap‑regret, which is o(T).

The most striking contribution concerns mean‑based no‑regret learners (Definition 2). Such algorithms rarely play actions whose cumulative reward lags far behind the best arm. Theorem 9 shows that if the learner has three or more actions, the optimizer can design a game and a dynamic policy that steers the learner’s cumulative reward vector so that the learner repeatedly selects suboptimal actions from the optimizer’s perspective. Consequently, the optimizer can achieve a total utility V′·T − o(T) with V′ > V, strictly surpassing the Stackelberg payoff. By contrast, Theorem 8 proves that with only two learner actions, even mean‑based learners cannot be exploited beyond V.

Finally, the paper formulates the optimizer’s optimal long‑run policy as an N‑dimensional control problem, where the state consists of the learner’s cumulative utilities for each of its N actions. The optimizer’s action at each round influences the state transition, and the objective is to maximize average payoff while respecting the mean‑based selection rule. The authors discuss structural simplifications (e.g., symmetry, dimensionality reduction) and leave the exact solution of this control problem as an open question, offering conjectures about its form.

Overall, the work establishes Stackelberg utility as a universal baseline against any no‑regret learner, identifies precise conditions (constant‑sum, two‑action learners, no‑swap‑regret) under which this baseline cannot be exceeded, and reveals that mean‑based learners with richer action sets admit strictly higher payoffs for the optimizer, thereby deepening our understanding of strategic exploitation of learning dynamics.