In this work, we suggest a parameterized statistical model (the gamma distribution) for the frequency of word occurrences in long strings of English text and use this model to build a corresponding thermodynamic picture by constructing the partition function. We then use our partition function to compute thermodynamic quantities such as the free energy and the specific heat. In this approach, the parameters of the word frequency model vary from word to word so that each word has a different corresponding thermodynamics and we suggest that differences in the specific heat reflect differences in how the words are used in language, differentiating keywords from common and function words. Finally, we apply our thermodynamic picture to the problem of retrieval of texts based on keywords and suggest some advantages over traditional information retrieval methods.
Deep Dive into Thermodynamics of Information Retrieval.
In this work, we suggest a parameterized statistical model (the gamma distribution) for the frequency of word occurrences in long strings of English text and use this model to build a corresponding thermodynamic picture by constructing the partition function. We then use our partition function to compute thermodynamic quantities such as the free energy and the specific heat. In this approach, the parameters of the word frequency model vary from word to word so that each word has a different corresponding thermodynamics and we suggest that differences in the specific heat reflect differences in how the words are used in language, differentiating keywords from common and function words. Finally, we apply our thermodynamic picture to the problem of retrieval of texts based on keywords and suggest some advantages over traditional information retrieval methods.
Let us imagine that we are looking for some article in the Web. Probably the first thing we will do is to go to a search engine and type some keywords.
If we type a query like “I am looking for an article about statistical mechanics of images”, although it is exactly what we want, we will probably get nothing related to the subject or we will get only a content partially related to it. In order to have some meaningful results, one needs to refine the query to something like “image”, “statistical mechanics” ignoring in this way the structure of the language and using some statistical estimations of the parts of the query that stick well with its meaning.
Current web search engines are a product of some 15 years evolution. This evolution has shown that if we are looking for the meaning of a text, we must look for specific, statistically salient keywords that are supposed to be present in it, largely ignoring the syntactic and the semantics structure of the language.
Probably, the best way to do the analysis of a text, written is some language [1], would be to have some exact descriptions of the language, for example, a weighted context-free grammar [2]. Having in mind the Zipf’s law [3] of the frequency distribution of the words, even if reasonable grammar exists, in a single text of arbitrary length we will have some 40% halomorphemes [4]. As a consequence, the length of the grammar will be of the order of the length of the text for any text we choose.
Therefore, it is convenient to consider the language as a set of all the texts spoken/written in that language. Using statistical arguments, we do not need all texts, but only a significantly large random set of texts in order to treat the problem.
In this article we propose a statistical physics model of the text that treats the text as a large random data set. The text is regarded to be conditioned on the language in which the text is written and can be restricted on the area to which it belongs, as for example “nonlinear physics” or “novels of 17 th cen-tury”.
The model we investigate consists of a text T and a vocabulary V , written in some language. The vocabulary is formed using as a basis some huge collection of texts, written in that language.
The relationship between the vocabulary and the text is asymmetric. If we regard an article of nonlinear science, it is highly probable to find words like “chaotic dynamics” or “Hamiltonian”, but highly improbable to find words like “horses” and “knights”. Regarding, for example “Don Quixote”, it is just the opposite. So a text that treats some subject is highly restricted by this subject and the later conditions the vocabulary used. The language as a whole has no such restriction. Therefore, the relative excess (or higher frequency) of a word in the vocabulary is a normal situation.
On the contrary, the relative excess of a word in the text has a specific meaning, because if the word is with much higher occurrence in the text than in the common language, that can be interpreted as an indication that this text treats exactly a subject expressed by this word, e.g. that the word is a specific term or keyword in the text. This is the first class of words in the text that we will consider in this article.
On the other hand, the text will always contain words that are common in the language, which have more or less the same frequency in any text and in the vocabulary. A large fraction of the words of that type will be formed by the so called function words. These words by themselves carry no meaning but are essential for expressing the language structure. A typical example of a function word in English is the word “the”. The problem with this category is that it is not very easy to define it in a way that can be implemented by a computer program. A similar and strictly defined category is the class of closed class words that by definition are the words, which do not change their form in any text.
Finally the third class of words that will follow more or less the same frequency distribution in the text and in the vocabulary are the common words. They serve to transmit the meaning of the text, but are common for every text that must explain some concept, like for example the word “explain” in this sentence. In this class significant deviations between different texts and different authors can be expected.
In the literature, the statistical treatment of the text is mainly regarded in relation with the information retrieval (IR) theory, where this consideration results very fruitful [5,6].
Another statistical consideration is centered on the Zipf law [3,7] and looks for the relative distribution of different words (types) in a collection of texts. The Zipf law can be derived from the requirement of maximal information exchange [8]. This approach mainly focuses on the tail of the distribution, that is an example of large number of rare events (LNRE).
In this article we fix the length of the text to some reason
…(Full text truncated)…
This content is AI-processed based on ArXiv data.
