Towards the quantification of the semantic information encoded in written language

Written language is a complex communication signal capable of conveying information encoded in the form of ordered sequences of words. Beyond the local order ruled by grammar, semantic and thematic structures affect long-range patterns in word usage. Here, we show that a direct application of information theory quantifies the relationship between the statistical distribution of words and the semantic content of the text. We show that there is a characteristic scale, roughly around a few thousand words, which establishes the typical size of the most informative segments in written language. Moreover, we find that the words whose contributions to the overall information is larger, are the ones more closely associated with the main subjects and topics of the text. This scenario can be explained by a model of word usage that assumes that words are distributed along the text in domains of a characteristic size where their frequency is higher than elsewhere. Our conclusions are based on the analysis of a large database of written language, diverse in subjects and styles, and thus are likely to be applicable to general language sequences encoding complex information.

💡 Research Summary

The paper tackles the longstanding problem of quantifying how much semantic information is embedded in written language by applying information‑theoretic measures directly to word distributions. The authors begin by treating a text as a sequence of symbols whose statistical properties are shaped not only by local grammatical constraints but also by long‑range semantic and thematic structures. To capture these effects, they partition each document into windows of a given length and compute, for each window, the Kullback‑Leibler divergence between the observed word frequencies and the frequencies expected from the whole corpus. This divergence serves as a proxy for the “information content” of the window, indicating how much the window deviates from a baseline, average language model.

By systematically varying the window size across a large, genre‑diverse corpus (including novels, scientific articles, newspaper reports, etc.), they discover a characteristic scale of roughly two to five thousand words at which the information content peaks. This scale suggests that the most informative segments of a text—those that carry the strongest semantic signal—tend to be of this size, aligning with cognitive theories that propose humans process text in thematic blocks of a few thousand words.

Beyond window‑level analysis, the authors introduce a word‑level metric: the cumulative contribution of each word to the total information across all windows. Words with the highest contributions are overwhelmingly content‑bearing terms (nouns, verbs, adjectives) directly related to the main topics of the document, whereas function words (articles, prepositions) contribute little. This observation leads to a simple generative model in which each word is distributed along the text in “domains” where its local frequency exceeds the global average. The typical domain size matches the previously identified window scale, providing a coherent explanation for the observed long‑range correlations.

The paper validates these findings across multiple domains, showing that the identified scale and the word‑information ranking are robust to variations in subject matter and writing style. Moreover, the authors compare their information‑based keyword extraction to traditional TF‑IDF and Latent Dirichlet Allocation (LDA) approaches, demonstrating that the information‑theoretic method more directly highlights the core concepts of a text without requiring external parameters or topic models.

In the discussion, the authors argue that this framework offers a new quantitative tool for a range of applications: automatic summarization, topic detection, semantic network construction, and even the evaluation of language models. They also outline future directions, such as extending the analysis to multilingual corpora, spoken language transcripts, and noisy social‑media data, as well as integrating information‑based features into machine‑learning pipelines for improved text understanding. Overall, the study provides compelling evidence that semantic information in written language is organized into characteristic, information‑dense segments, and that information theory can serve as a principled bridge between statistical word patterns and underlying meaning.