Semantic Chunking and the Entropy of Natural Language

The entropy rate of printed English is famously estimated to be about one bit per character, a benchmark that modern large language models (LLMs) have only recently approached. This entropy rate implies that English contains nearly 80 percent redundancy relative to the five bits per character expected for random text. We introduce a statistical model that attempts to capture the intricate multi-scale structure of natural language, providing a first-principles account of this redundancy level. Our model describes a procedure of self-similarly segmenting text into semantically coherent chunks down to the single-word level. The semantic structure of the text can then be hierarchically decomposed, allowing for analytical treatment. Numerical experiments with modern LLMs and open datasets suggest that our model quantitatively captures the structure of real texts at different levels of the semantic hierarchy. The entropy rate predicted by our model agrees with the estimated entropy rate of printed English. Moreover, our theory further reveals that the entropy rate of natural language is not fixed but should increase systematically with the semantic complexity of corpora, which are captured by the only free parameter in our model.

💡 Research Summary

The paper “Semantic Chunking and the Entropy of Natural Language” tackles a long‑standing puzzle in information theory: why printed English exhibits an entropy rate of roughly one bit per character, implying about 80 % redundancy relative to a random five‑bit‑per‑character baseline. While modern large language models (LLMs) can now empirically estimate this rate, no first‑principles account has explained its origin. The authors propose a statistical framework that models natural language as a hierarchy of semantically coherent “chunks” obtained by recursively segmenting a text. At each recursion level the text is split into at most K contiguous chunks; the process continues until every leaf is a single token. This yields a K‑ary tree, termed a “semantic tree,” whose internal nodes correspond to increasingly coarse semantic spans and whose leaves are individual tokens.

The key theoretical contribution is the mapping of the ensemble of semantic trees onto a random K‑ary tree model. For a parent chunk of size n, the size m of any child is drawn from a distribution p_split(m|n)=Z_{K‑1}(n‑m)/Z_K(n), where Z_K(n)=C(n+K‑1, K‑1) normalizes the combinatorial possibilities. This defines a Markov chain over chunk sizes and, crucially, leads to a closed‑form scaling law for the distribution of chunk sizes at any fixed depth L when the total token count N is large: P_L(n)≈(1/N) f_L(n/N). The scaling function f_L is obtained recursively via multiplicative convolution with a Beta(1, K‑1) distribution, reflecting the uniform random placement of split boundaries.

From the random‑tree ensemble the authors derive an analytic expression for the entropy per token, h_K = (1/⟨N⟩) E