Berezinskii--Kosterlitz--Thouless transition in a context-sensitive random language model

Several power-law critical properties involving different statistics in natural languages – reminiscent of scaling properties of physical systems at or near phase transitions – have been documented for decades. The recent rise of large language models has added further evidence and excitement by providing intriguing similarities with notions in physics such as scaling laws and emergent abilities. However, specific instances of classes of generative language models that exhibit phase transitions, as understood by the statistical physics community, are lacking. In this work, inspired by the one-dimensional Potts model in statistical physics, we construct a simple probabilistic language model that falls under the class of context-sensitive grammars, which we call the context-sensitive random language model, and numerically demonstrate an unambiguous phase transition in the framework of a natural language model. We explicitly show that a precisely defined order parameter – that captures symbol frequency biases in the sentences generated by the language model – changes from strictly zero to a strictly nonzero value (in the infinite-length limit of sentences), implying a mathematical singularity arising when tuning the parameter of the stochastic language model we consider. Furthermore, we identify the phase transition as a variant of the Berezinskii–Kosterlitz–Thouless (BKT) transition, which is known to exhibit critical properties not only at the transition point but also in the entire phase. This finding leads to the possibility that critical properties in natural languages may not require careful fine-tuning nor self-organized criticality, but are generically explained by the underlying connection between language structures and the BKT phases.

💡 Research Summary

The paper presents a novel probabilistic language model that exhibits a genuine phase transition of the Berezinskii‑Kosterlitz‑Thouless (BKT) type, thereby providing a concrete bridge between statistical‑physics concepts and linguistic phenomena. The authors begin by reviewing the long‑standing observation that natural languages obey power‑law regularities such as Zipf’s law and that recent large language models (LLMs) display scaling laws and emergent abilities reminiscent of critical phenomena. They note that previous attempts to locate phase transitions in probabilistic context‑free grammars (CFGs) have failed, largely because CFGs lack the expressive power to generate long‑range correlations.

To overcome this limitation, the authors construct a Context‑Sensitive Random Language Model (CSRLM), a probabilistic grammar situated one level higher in the Chomsky hierarchy (context‑sensitive grammars, CSGs). The model is inspired by the one‑dimensional long‑range Potts (or “Pott’s”) model. Its dynamics consist of three interacting processes:

1. Growth – a non‑terminal symbol X expands via a rule X → Y Z, increasing the string length. A growth parameter q controls the balance between expansion and rewriting, allowing the system to reach a thermodynamic limit as sentence length L → ∞.
2. Context‑Sensitive Rewrites – a substring α‑X‑α′ may be rewritten as α‑Y‑α′ with Metropolis acceptance probability p = min(1, e^{−ΔE/k_BT}). The energy ΔE is defined by a pairwise coupling of symbols that can be long‑ranged; temperature T therefore governs how likely a rule is applied.
3. Termination – non‑terminals may become terminals, after which they no longer evolve. For analytical simplicity this step is omitted, so the model’s size grows deterministically.

The authors define an order parameter m as the relative frequency (magnetization) of a chosen symbol in the generated string. They compute its mean, variance (susceptibility χ = L Var(m)), and the Binder cumulant U = ⟨m⁴⟩/⟨m²⟩², which is the normalized kurtosis. In a disordered phase U → 0, in a conventional ordered phase U → 1, while a BKT transition yields a non‑trivial value between 0 and 1 that varies smoothly with temperature.

Extensive Monte‑Carlo simulations are performed for binary alphabets (K = 2) and for larger alphabets (K > 2). System sizes up to L ≈ 2¹⁴ are examined, and finite‑size scaling is applied. The results show:

• For low temperatures (T < T_c) the magnetization becomes non‑zero, susceptibility diverges as χ ∝ L^{2−η} with η ≈ 0.25, and the correlation function decays as a power law C(r) ∝ r^{−η}. This is precisely the hallmark of a BKT phase.
• The Binder cumulant exhibits a smooth crossover rather than a step function, confirming the infinite‑order nature of the transition.
• Remarkably, the BKT‑type behavior appears even when the interaction exponent s = 0 (i.e., distance‑independent coupling), whereas the traditional 1‑D long‑range Potts model requires s = 1 for a BKT transition. This suggests that the growth‑rewrite mechanism of the language model creates effective long‑range correlations beyond the static interaction kernel.
• Multi‑letter alphabets also display the same qualitative behavior, indicating that the phenomenon is robust to the size of the symbol set.

The authors argue that because a BKT phase is critical throughout an entire parameter region, the scaling laws observed in natural language and LLMs need not be the result of fine‑tuned critical points or self‑organized criticality. Instead, the underlying grammar‑induced long‑range correlations can place the system naturally within a critical regime. Consequently, the order parameter and Binder cumulant provide a quantitative way to distinguish “meaningful” language generation (ordered/BKT phase) from “gibberish” (disordered phase).

In the discussion, the paper highlights the broader implications: (i) a concrete example of a one‑dimensional system exhibiting a BKT transition, (ii) a potential universal explanation for robust linguistic scaling laws, and (iii) a new analytical toolbox for assessing generative language models. The authors acknowledge that the CSRLM is a highly simplified toy model, but they suggest that extending the framework to incorporate realistic token vocabularies, training dynamics, and neural architectures could yield deeper insights into why large language models display emergent abilities and power‑law scaling without meticulous hyperparameter tuning.

Overall, the work convincingly demonstrates that a carefully designed context‑sensitive probabilistic grammar can undergo a BKT transition, thereby offering a physics‑based perspective on linguistic criticality and opening avenues for future interdisciplinary research between statistical physics and modern AI language modeling.