Scale-Dependent Semantic Dynamics Revealed by Allan Deviation

While language progresses through a sequence of semantic states, the underlying dynamics of this progression remain elusive. Here, we treat the semantic progression of written text as a stochastic trajectory in a high-dimensional state space. We utilize Allan deviation, a tool from precision metrology, to analyze the stability of meaning by treating ordered sentence embeddings as a displacement signal. Our analysis reveals two distinct dynamical regimes: short-time power-law scaling, which differentiates creative literature from technical texts, and a long-time crossover to a stability-limited noise floor. We find that while large language models successfully mimic the local scaling statistics of human text, they exhibit a systematic reduction in their stability horizon. These results establish semantic coherence as a measurable physical property, offering a framework to differentiate the nuanced dynamics of human cognition from the patterns generated by algorithmic models.

💡 Research Summary

The paper introduces a novel quantitative framework for analyzing how the meaning of a text evolves over the course of its sentences. By converting each sentence into a fixed‑dimensional embedding (using a sentence‑transformer such as all‑MiniLM‑L6‑v2) and measuring the cosine distance between successive sentences, the authors construct a one‑dimensional “semantic phase” signal: a cumulative sum of incremental semantic displacements indexed by sentence number. This signal preserves the temporal order of the narrative while discarding absolute semantic coordinates, allowing the dynamics of meaning change to be studied directly.

To probe scale‑dependent fluctuations of this signal, the authors apply Allan deviation, a variance estimator originally developed for precision metrology. Allan deviation σ(τ) quantifies how the difference between adjacent averages of the signal evolves with the averaging scale τ (the number of sentences used for coarse‑graining). In a log‑log plot, a straight‑line region indicates power‑law scaling σ(τ) ∝ τ^α, where α = –0.5 corresponds to a memoryless random walk (white noise).

The study examines a broad corpus spanning creative literature (novels, drama, epic poetry, short stories) and technical or informational genres (biology, chemistry, physics, mathematics, encyclopedic entries). Across all texts, two distinct regimes emerge:

1. Short‑time scaling (τ up to ≈5 % of the total length).
   All corpora display clear power‑law behavior, but the exponent α differs systematically by genre. Creative works cluster around α ≈ –0.4, indicating that successive sentences are nearly uncorrelated in their semantic shift—reflecting a high degree of local freedom in meaning exploration. Technical texts show flatter exponents (α ≈ –0.25 to –0.30), signifying stronger local correlations that preserve conceptual consistency. Randomizing sentence order eliminates these differences, confirming that the observed scaling arises from ordered semantic progression rather than embedding geometry or text length.

2. Long‑time crossover to a noise floor.
   Beyond a certain τ, σ(τ) flattens, defining a “noise floor” where additional averaging no longer reduces variance. The authors operationalize this transition as a “context horizon”: the smallest τ at which the short‑time slope deviates by more than 15 % from its initial value. Context horizons vary dramatically across genres. Creative narratives often lack a detectable crossover within the measured range, implying scale‑invariant semantic organization. In contrast, technical and encyclopedic texts reach the noise floor after only a few percent of their total length, indicating rapid convergence to a limited semantic space.

The framework is shown to be robust to the choice of embedding model; alternative transformer‑based sentence embeddings yield quantitatively consistent short‑time exponents. Moreover, the method is insensitive to absolute distance values or embedding dimensionality because Allan deviation only examines the scaling of cumulative increments.

A key contribution is the comparison between human‑written texts and outputs from several state‑of‑the‑art large language models (Claude 3 Opus, GPT‑4 Turbo 2024, Llama‑3 70B, GigaChat Pro, YandexGPT 3 Pro) generated under identical prompts. All models reproduce the human‑like short‑time exponent (α ≈ –0.38 ~ –0.41), demonstrating that current generative systems capture sentence‑to‑sentence semantic variability. However, at larger τ the models deviate from the short‑time regime much earlier than humans: their context horizons occur after only 13–26 sentences, whereas human texts maintain scaling for 30–40 sentences or more (often exceeding 5 % of the text length). This suggests that contemporary autoregressive models quickly converge to a semantic “core” and lack the capacity to continuously explore new semantic territories, a limitation that is quantitatively exposed by the Allan‑deviation analysis.

Methodologically, the approach offers several advantages. By collapsing high‑dimensional embeddings into a one‑dimensional cumulative signal, it sidesteps the curse of dimensionality while retaining temporal information. Allan deviation provides a clean separation between short‑term variability and long‑term drift, something that traditional power‑spectral or detrended fluctuation analyses cannot achieve without discarding order information. The random‑order control experiments further validate that the observed scaling is genuinely dynamical.

In the discussion, the authors interpret the steeper short‑time scaling of creative literature as evidence of “semantic freedom” – writers can introduce diverse ideas without strong immediate constraints, yet still maintain coherence through long‑range structure. Technical writing, by contrast, exhibits stronger local correlations to ensure precision and logical flow. The context horizon is proposed as a physically interpretable metric of semantic stability, offering a new lens for genre classification, authorship attribution, and evaluation of generative models.

Overall, the paper bridges statistical physics and computational linguistics by repurposing a metrological tool to quantify meaning dynamics. It establishes Allan deviation as a language‑agnostic, model‑agnostic instrument for probing the scale‑dependent organization of text, revealing fundamental differences between human cognition and current AI generation. The findings open avenues for designing next‑generation language models that better emulate the long‑range semantic stability characteristic of human writing.