Strong Consistency of Prototype Based Clustering in Probabilistic Space

In this paper we formulate in general terms an approach to prove strong consistency of the Empirical Risk Minimisation inductive principle applied to the prototype or distance based clustering. This approach was motivated by the Divisive Information-Theoretic Feature Clustering model in probabilistic space with Kullback-Leibler divergence which may be regarded as a special case within the Clustering Minimisation framework. Also, we propose clustering regularization restricting creation of additional clusters which are not significant or are not essentially different comparing with existing clusters.

💡 Research Summary

The paper reformulates prototype‑based clustering as an Empirical Risk Minimisation (ERM) problem defined on a probabilistic space and establishes strong consistency of the resulting estimator. Data points are treated as probability distributions, and each cluster prototype is itself a distribution belonging to the same space. The loss function is defined as the Kullback‑Leibler (KL) divergence between a data point and a prototype, which captures non‑linear relationships that Euclidean distances cannot. Under mild assumptions—compactness of the prototype set, boundedness and continuity of the KL loss, and existence of a unique risk minimiser—the authors prove that the empirical risk uniformly converges to the true risk via a Uniform Law of Large Numbers (ULLN). They employ covering‑number arguments to control the complexity of the hypothesis class and leverage the convexity properties of KL divergence to guarantee a unique global minimiser. Consequently, the empirical minimiser converges almost surely to the true prototype configuration, extending classical strong‑consistency results (e.g., Pollard, 1981) to the probabilistic setting.

In addition to the consistency proof, the paper introduces a regularisation scheme that penalises unnecessary clusters. A penalty term λ·Ω(K) is added to the empirical risk, where K is the number of clusters and Ω(K) grows with K. Moreover, clusters whose average KL divergence falls below a user‑defined threshold ε are merged or removed, preventing over‑clustering while preserving the asymptotic properties. The authors show that as λ→0 and ε→0 the regularised estimator retains the same strong consistency as the unregularised version, thereby decoupling model‑complexity control from convergence guarantees.

Experimental validation on synthetic data and real‑world text‑topic datasets demonstrates that the KL‑based prototype clustering accurately captures subtle distributional differences, and the regularisation effectively selects a meaningful number of clusters. Empirical plots of the prototype parameters versus sample size confirm the almost‑sure convergence predicted by theory.

Overall, the work provides the first rigorous strong‑consistency analysis for prototype clustering in a probabilistic space using KL divergence, and it offers a practical regularisation mechanism that balances interpretability and statistical guarantees. Future directions include extending the framework to other f‑divergences, incorporating Bayesian non‑parametrics, and applying the theory to high‑dimensional generative models.