Verification of the Implicit World Model in a Generative Model via Adversarial Sequences

Generative sequence models are typically trained on sample sequences from natural or formal languages. It is a crucial question whether – or to what extent – sample-based training is able to capture the true structure of these languages, often referred to as the ``world model’’. Theoretical results indicate that we can hope for soundness at best, that is, generating valid sequences, but not necessarily all of them. However, it is still important to have practical tools that are able to verify whether a given sequence model is sound. In this study, we focus on chess, as it is a domain that provides enough complexity while having a simple rule-based world model. We propose adversarial sequence generation for verifying the soundness of the sequence model. Our adversaries generate valid sequences so as to force the sequence model to generate an invalid next move prediction. Apart from the falsification of soundness, this method is also suitable for a more fine-grained analysis of the failure modes and the effects of different choices during training. To demonstrate this, we propose a number of methods for adversarial sequence generation and evaluate the approach on a large set of chess models. We train models on random as well as high-quality chess games, using several training recipes. We find that none of the models are sound, but some training techniques and dataset choices are able to improve soundness remarkably. We also investigate the potential application of board state probes in both our training and attack methods. Our findings indicate that the extracted board states have no causal role in next token prediction in most of the models.

💡 Research Summary

The paper investigates whether generative sequence models trained on samples of a formal language actually learn the underlying “world model” that governs the language. Using chess as a testbed—because its rules constitute a clear, deterministic world model—the authors propose an adversarial verification framework that seeks to falsify the soundness of a model. Soundness is defined formally: a model is sound if, for every valid prefix of moves, the model’s decoding policy selects a next move that is also valid according to the true world model.

To test this, the authors construct adversarial sequences that are themselves always legal, but are chosen so that the model is forced to predict an illegal continuation at some later step. The adversary operates by extending a valid prefix a₁…a_k with a move a_{k+1} that maximizes an auxiliary function f(M, a₁…a_k a_{k+1}). Several concrete instantiations of f are explored:

1. Illegal Move Oracle (IMO) – selects the legal move that maximally increases the probability that the model will output an illegal move next.
2. Board State Oracle (BSO) – uses a linear probe that predicts the board configuration; the adversary picks the move that maximizes the probe’s loss, testing the hypothesis that board‑state predictions causally affect next‑token decisions.
3. Adversarial Detours (AD) – follows the approach of Vafa et al. (2024) by picking the legal move with the lowest model probability, thereby pushing the model toward out‑of‑distribution regions.
4. Random Move (RM) – a non‑informed baseline that picks a legal move uniformly at random.
5. Sequence Model Move (SMM) – a benevolent baseline where the adversary simply lets the model choose its highest‑probability legal move.

The authors train a suite of models under different data regimes and objectives. Datasets include high‑quality human games (MB‑500k), engine‑generated games (Stockfish‑8M), a large corpus of human games from Lichess (16M), and several random‑game collections (500 K to 10 M games). Two training objectives are used: the standard next‑token (NT) prediction and a probability‑distribution (PD) objective that forces the model to assign a uniform distribution over all legal moves at each step, thereby encouraging a more explicit representation of the rule set. Tokenization follows the scheme of Toshnial et al. (2022), encoding each square and promotion piece as a single token, so moves occupy 2–3 tokens.

The experimental protocol pits each model against each adversary in a two‑player setting where the adversary always plays White and the model replies as Black. An attack succeeds if the model ever produces an illegal move. Results show that all models are unsound: each can be forced to make an illegal move by at least one adversarial strategy. IMO is the most effective attack, achieving success rates often above 70 %, while BSO is considerably weaker (≈30 %). AD and RM rarely succeed (<15 %). The SMM baseline never causes a violation, confirming that the attacks are indeed adversarial.

Dataset choice matters but not dramatically. Models trained on random games tend to be slightly more robust to IMO than those trained on curated human games, supporting prior observations that randomness can aid rule learning. The PD objective yields modest improvements in soundness (5–10 % higher success rates) compared to pure NT training, suggesting that exposing the model to the full legal move distribution helps but does not solve the problem.

Crucially, the BSO attack’s limited success indicates that the linear board‑state probes, despite achieving high classification accuracy, do not have a causal influence on the model’s next‑move predictions. The loss of the probe does not correlate with the likelihood of an illegal move, challenging the assumption that probing alone can certify the presence of a correct world model.

The paper contributes a practical, scalable method for existential falsification of world‑model soundness without needing to define a probability threshold for the generated language, as required by prior work. It also provides a detailed analysis of failure modes across training regimes, datasets, and attack designs. Limitations include the focus on chess (a relatively simple deterministic game) and the reliance on linear probes; extending to more complex games, programming languages, or richer probing mechanisms remains future work.

In conclusion, the study demonstrates that current generative sequence models, even when trained on massive high‑quality chess data, do not reliably internalize the full rule set of the domain. Adversarial sequence generation offers a clear, interpretable way to expose these gaps, and the findings call for new training objectives, possibly adversarial training, and more causally grounded probing techniques to move toward truly sound world models.