Information theoretic model validation for clustering

Model selection in clustering requires (i) to specify a suitable clustering principle and (ii) to control the model order complexity by choosing an appropriate number of clusters depending on the noise level in the data. We advocate an information theoretic perspective where the uncertainty in the measurements quantizes the set of data partitionings and, thereby, induces uncertainty in the solution space of clusterings. A clustering model, which can tolerate a higher level of fluctuations in the measurements than alternative models, is considered to be superior provided that the clustering solution is equally informative. This tradeoff between \emph{informativeness} and \emph{robustness} is used as a model selection criterion. The requirement that data partitionings should generalize from one data set to an equally probable second data set gives rise to a new notion of structure induced information.

💡 Research Summary

The paper tackles the long‑standing problem of model selection in clustering from an information‑theoretic standpoint. Traditional approaches usually focus on a single criterion—such as silhouette score, Bayesian Information Criterion (BIC), or Gap Statistic—to decide both the clustering principle and the number of clusters. However, these methods often fail when measurement noise is substantial, leading to over‑ or under‑estimation of the true cluster structure.
The authors propose to view the data acquisition process as a quantization of the continuous measurement space: because any real‑world sensor records values with finite precision, the set of possible data partitions (or “clusterings”) is itself quantized into a finite codebook. This quantization induces an entropy over the space of partitions, which grows with the noise level. Two complementary quantities are then defined for any candidate clustering model:

1. Informativeness – the reduction in data entropy achieved by the partition, i.e., how much structure the clustering extracts from the data. Formally, it is the mutual information (I(\Pi;X)=H(X)-H(X|\Pi)).
2. Robustness – the probability that the same partition persists when the measurements are perturbed by additional noise. This captures the model’s tolerance to fluctuations and is measured by the stability of the partition under increasing noise amplitudes.

The central thesis is that a clustering model that tolerates higher measurement fluctuations while delivering the same amount of information should be preferred. This trade‑off between informativeness and robustness becomes the new model‑selection criterion. Mathematically, the optimal model (\mathcal{M}^*) is obtained by maximizing a weighted sum (or Pareto frontier) of the two quantities: \