A Short Introduction to Model Selection, Kolmogorov Complexity and Minimum Description Length (MDL)

The concept of overfitting in model selection is explained and demonstrated with an example. After providing some background information on information theory and Kolmogorov complexity, we provide a short explanation of Minimum Description Length and error minimization. We conclude with a discussion of the typical features of overfitting in model selection.

💡 Research Summary

The paper addresses the pervasive problem of over‑fitting in model selection by grounding the discussion in information theory and algorithmic complexity. It begins with a clear definition of over‑fitting: a model that captures not only the underlying signal but also the random noise in the training data, leading to a dramatic drop in performance on validation or real‑world data. Traditional remedies—cross‑validation, regularization, early stopping—are acknowledged, but the authors argue that these techniques rely heavily on empirical tuning and lack a solid theoretical foundation.

To provide that foundation, the authors introduce Kolmogorov complexity, the length of the shortest program that can generate a given data string. Although Kolmogorov complexity is uncomputable in practice, it serves as the conceptual ideal for “minimum information content.” The paper then moves to the Minimum Description Length (MDL) principle, which operationalizes Kolmogorov’s idea by seeking the model that minimizes the total code length required to describe both the model itself (its parameters) and the data given the model. This total description length is split into two components: the model complexity term (bits needed to encode the model) and the data‑fit term (bits needed to encode the residuals or errors).

Two concrete MDL formulations are examined. The first, two‑part MDL, encodes the parameters first and then the residuals. The second, Normalized Maximum Likelihood (NML), normalizes over all possible data sequences, yielding a universal code length that is optimal in a minimax sense. While both approaches penalize unnecessary complexity, NML’s exhaustive normalization makes it computationally prohibitive for all but the simplest model families.

The authors illustrate the inherent trade‑off between error minimization and description‑length minimization with graphs and equations. As model complexity increases, the training error falls but the description length rises; conversely, forcing a short description inflates the error. This trade‑off mirrors classic information criteria such as AIC and BIC, which the paper shows are special cases of MDL with particular choices of penalty terms (AIC uses a linear penalty on the number of parameters, BIC uses a logarithmic penalty based on sample size).

A particularly valuable contribution is the systematic enumeration of four hallmark symptoms of over‑fitting: (1) the “error reversal” where training error continues to drop while validation error climbs, (2) an explosion in the number of parameters relative to the number of observations, (3) the model’s tendency to learn noise, creating unnecessarily intricate structures, and (4) a sharp degradation in generalization performance on new data. The paper argues that an MDL‑based selection criterion naturally curtails each of these symptoms because it explicitly balances model complexity against data fidelity.

Practical implementation steps are outlined: (i) define a candidate model set, (ii) compute for each candidate the code length of its parameters and the code length of its residuals, (iii) sum these to obtain the total description length, (iv) select the model with the smallest total, and (v) optionally validate the choice with cross‑validation to guard against implementation approximations. This workflow integrates the theoretical rigor of MDL with the empirical safety net of traditional validation techniques.

In conclusion, the paper successfully bridges algorithmic information theory and statistical model selection. By framing over‑fitting as a violation of the principle of shortest total description, it provides both a deep theoretical justification for penalizing complexity and a concrete, actionable methodology for practitioners in statistics, machine learning, and data science. The discussion of Kolmogorov complexity, MDL variants, and the explicit connection to AIC/BIC makes the work a valuable reference for researchers seeking a principled approach to avoid over‑fitting while maintaining predictive accuracy.