Kolmogorov Complexity in perspective. Part I: Information Theory and Randomnes

We survey diverse approaches to the notion of information: from Shannon entropy to Kolmogorov complexity. Two of the main applications of Kolmogorov complexity are presented: randomness and classification. The survey is divided in two parts in the same volume. Part I is dedicated to information theory and the mathematical formalization of randomness based on Kolmogorov complexity. This last application goes back to the 60’s and 70’s with the work of Martin-L"of, Schnorr, Chaitin, Levin, and has gained new impetus in the last years.

💡 Research Summary

The paper provides a comprehensive survey of the evolution of the concept of information, tracing its development from Shannon’s probabilistic entropy to Kolmogorov’s algorithmic complexity. It begins by outlining the strengths and limitations of Shannon entropy, emphasizing its role in quantifying average information content for a given probability distribution and its centrality to coding theory and data compression. However, Shannon’s framework does not assign a complexity measure to individual strings, a gap that Kolmogorov complexity (K) fills by defining the information content of a specific object as the length of the shortest program that produces it on a universal Turing machine.

The core of the manuscript is devoted to the formalization of randomness using Kolmogorov complexity. It reviews the seminal contributions of Martin‑Löf, Schnorr, Chaitin, and Levin. Martin‑Löf introduced the notion of effective statistical tests, defining a sequence as random if it passes all computably enumerable tests of measure zero. Schnorr’s approach focuses on computable betting strategies, while Chaitin’s definition directly ties randomness to incompressibility: a string x is random if K(x) ≥ |x| – c for some constant c. Levin’s K‑theorem links algorithmic complexity with probability, showing that strings of high complexity correspond to low-probability events. The paper highlights Chaitin’s Ω number as a concrete example of a maximally random real whose binary expansion is algorithmically random, illustrating the deep connection between randomness, halting probabilities, and incompleteness.

Beyond theory, the authors discuss two major applications of Kolmogorov complexity. First, randomness testing: traditional statistical batteries (e.g., NIST tests) assess the distributional properties of pseudo‑random generators, whereas complexity‑based tests evaluate the compressibility of the output directly, offering a more intrinsic measure of unpredictability. Second, classification and clustering: the Normalized Compression Distance (NCD), derived from approximations of K using real-world compressors, provides a universal, parameter‑free metric for comparing objects ranging from texts to genomes. Recent work integrating NCD with machine‑learning pipelines has demonstrated superior performance in unsupervised learning tasks, especially when structural similarity is not captured by conventional distance measures.

The paper concludes by identifying open research directions. Since exact Kolmogorov complexity is uncomputable, developing tighter, provably efficient approximations remains a priority. Extending the randomness framework to quantum computational models and exploring its implications for cryptographic key generation are highlighted as promising avenues. Moreover, applying algorithmic information theory to biological sequences, network data, and other high‑dimensional domains could yield novel insights into the inherent complexity of natural systems.

Overall, the manuscript argues convincingly that Kolmogorov complexity is not merely a theoretical curiosity but a foundational tool that bridges information theory, randomness, and practical data analysis, offering a unified perspective that continues to inspire new research across computer science, mathematics, and the natural sciences.