A statistical mechanical interpretation of instantaneous codes

In this paper we develop a statistical mechanical interpretation of the noiseless source coding scheme based on an absolutely optimal instantaneous code. The notions in statistical mechanics such as statistical mechanical entropy, temperature, and thermal equilibrium are translated into the context of noiseless source coding. Especially, it is discovered that the temperature 1 corresponds to the average codeword length of an instantaneous code in this statistical mechanical interpretation of noiseless source coding scheme. This correspondence is also verified by the investigation using box-counting dimension. Using the notion of temperature and statistical mechanical arguments, some information-theoretic relations can be derived in the manner which appeals to intuition.

💡 Research Summary

The paper presents a novel statistical‑mechanical framework for analyzing noiseless source coding based on an absolutely optimal instantaneous code. An instantaneous code assigns a binary codeword to each source symbol such that the codewords can be decoded on the fly without ambiguity. When the code is absolutely optimal, the length ℓ(x) of the codeword for symbol x satisfies ℓ(x)=−log p(x) (with logarithms taken in base 2), and the expected code length equals the Shannon entropy H(X). The authors reinterpret these quantities in the language of statistical mechanics.

First, they treat each possible codeword length ℓ(x) as a microscopic energy level E. The probability of a symbol under a “temperature” T is then defined by the Boltzmann‑like distribution

 p_T(x)=exp