Modeling Next-Token Prediction as Left-Nested Intuitionistic Implication

We introduce the \emph{Arrow Language Model}, a neural architecture derived from an intuitionistic-logic interpretation of next-token prediction. Instead of representing tokens as additive embeddings mixed by attention, we encode a prefix as a \emph{left-nested implication chain} whose structure preserves order through non-commutative composition. Next-token prediction corresponds to \emph{modus ponens}, and sequence processing becomes constructive proof extension under the Curry–Howard correspondence. Our Prolog-based specialized theorem provers validate fundamental properties of the neural models, among which relations between commutative vs. non-commutative sequencing and single-token vs. multi-token prediction choices. We show that a neural architecture equivalent to multiplicative RNNs arises naturally from a proof-theoretic interpretation of next-token prediction as nested intuitionistic implication, we present a practical low-rank neural realization and position the model relative to Transformers and state-space models. Keywords: logic-based derivation of neural architectures, intuitionistic implicational logic, token-as-operator neural models, state-space models, alternatives to transformer-based foundational models.

💡 Research Summary

The paper proposes a novel neural architecture called the Arrow Language Model (ALM) that reinterprets next‑token prediction as a constructive proof step in intuitionistic propositional logic, specifically using only implication. The authors encode a token sequence w₁,…,wₙ as a left‑nested implication chain Lₙ = ((((w₁→w₂)→w₃)→… )→wₙ). This representation is inherently non‑commutative: permuting tokens changes the type of the overall implication, thereby preserving order without any positional encodings.

To ground this idea, the authors implement two Prolog theorem provers. The first, iprove, realizes Dyckhoff’s LJT sequent calculus for the implicational fragment, providing a sound and complete decision procedure that uses only modus ponens. The second, lprove, augments iprove with Curry‑Howard term construction, yielding λ‑terms that serve as proof witnesses for each provable sequent. These tools allow the authors to verify that a prefix implication Iₚ together with a full implication I_f = (Iₚ→w) yields the next token w via a single modus‑ponens step, i.e., I_f → (Iₚ→I_f) → w.

The logical view translates directly into a recurrent neural computation. Each token is treated as an operator fₜ that transforms the current hidden state sₜ (the “proof state”) into sₜ₊₁ = fₜ(sₜ). The operator corresponds to the implication “→” applied to the current prefix; applying the operator is analogous to β‑reduction of a λ‑term. The authors prove that the depth of the left‑nested chain (n − 1) matches exactly the number of recurrent steps required to evaluate or extend the proof, establishing a one‑to‑one correspondence between implication depth and recurrence depth.

A series of logical theorems are presented to illustrate how single‑step and multi‑step predictions can be expressed within the same framework. For example, the Curry‑Howard equivalents of the K and S combinators (p→q→p and (p→q→r)→(p→q)→p→r) are shown as valid implicational formulas. A key theorem ((p→q)→r) → (p→q→r) demonstrates a currying transformation that bridges a left‑nested “block‑to‑block” view and a right‑nested “token‑by‑token” view. Additional theorems illustrate how a rich contextual premise E can be factored out, leaving a smaller local trigger p to derive the next token, and how multi‑token continuations can be compiled from a fixed context into a single higher‑order implication. These results suggest that a language model can internally compile a prefix into reusable continuation operators, enabling efficient multi‑token generation.

From an implementation standpoint, the authors map each token to a low‑rank linear operator (e.g., a matrix or tensor) and realize implication composition as non‑commutative matrix multiplication. By sharing parameters and employing low‑rank approximations, the model achieves a parameter count comparable to multiplicative RNNs. Empirical comparisons show that the ALM matches the performance of multiplicative RNNs while offering a clear logical interpretation absent in standard Transformers. Moreover, the model’s reliance on explicit implication composition provides a transparent alternative to the implicit order encoding of self‑attention.

Beyond generation, the paper leverages the Prolog provers for logical information retrieval. Sentences from a corpus are converted into left‑nested implication formulas and stored as dynamic Prolog facts. The predicates ipref, isuff, and isufpref systematically generate all prefixes, suffixes, and prefix‑of‑suffix sub‑formulas, enabling queries that retrieve original sentences matching a given sub‑formula. This demonstrates that logical proof search can serve as a principled memory‑lookup mechanism, tightly coupling the model’s learned knowledge with a symbolic retrieval process.

In summary, the work reframes language modeling from a probabilistic next‑token distribution problem to a constructive proof‑construction problem in intuitionistic implication logic. It provides a complete pipeline: logical encoding of sequences, formal verification via Prolog, a neural realization that mirrors proof steps, and a logical retrieval system. The approach offers interpretability, explicit order handling, and a theoretical bridge between symbolic reasoning and neural sequence models. Future directions include extending the framework to richer logics (linear, modal), scaling the architecture to large corpora, and integrating logical retrieval with generation in a unified system.