Benchmarking of Clustering Validity Measures Revisited

Validation plays a crucial role in the clustering process. Many different internal validity indexes exist for the purpose of determining the best clustering solution(s) from a given collection of candidates, e.g., as produced by different algorithms or different algorithm hyper-parameters. In this study, we present a comprehensive benchmark study of 26 internal validity indexes, which includes highly popular classic indexes as well as more recently developed ones. We adopted an enhanced revision of the methodology presented in Vendramin et al. (2010), developed here to address several shortcomings of this previous work. This overall new approach consists of three complementary custom-tailored evaluation sub-methodologies, each of which has been designed to assess specific aspects of an index’s behaviour while preventing potential biases of the other sub-methodologies. Each sub-methodology features two complementary measures of performance, alongside mechanisms that allow for an in-depth investigation of more complex behaviours of the internal validity indexes under study. Additionally, a new collection of 16177 datasets has been produced, paired with eight widely-used clustering algorithms, for a wider applicability scope and representation of more diverse clustering scenarios.

💡 Research Summary

The paper presents a comprehensive benchmark of 26 internal clustering validity indexes using an unprecedented collection of 16 177 datasets and eight widely used clustering algorithms. Building on the methodology of Vendramin et al. (2010), the authors identify several shortcomings of the earlier work—most notably its reliance on a single “optimal” partition, the exclusive use of Pearson correlation, and a limited variety of data and algorithms. To overcome these issues, they propose a three‑pronged evaluation framework, each with two complementary performance measures.

Scenario 1 (Varied Number of Clusters) tests whether an index can correctly identify the true number of clusters when candidate solutions span a wide range of k values. Scenario 2 (Fixed Number of Clusters) fixes k to the ground‑truth value and evaluates how well the index discriminates high‑quality from low‑quality partitions. Scenario 3 (Algorithm‑ and External‑Index‑Independent) removes any dependence on a particular clustering algorithm or external validation measure, thereby assessing the pure internal capability of each index. For each scenario the authors compute (i) the proportion of correctly selected optimal partitions and (ii) the correlation between the internal index scores and three external indices (Jaccard, Adjusted Rand, Normalized Mutual Information).

Crucially, the study goes beyond linear Pearson correlation. It also reports Spearman’s ρ, Kendall’s τ, and visualizes the relationships with scatter plots and locally weighted regression curves. This reveals many non‑linear or piecewise patterns that Pearson alone would miss. For example, the Variance Ratio Criterion (VRC) shows a two‑region relationship with Jaccard, while the Ratkowski‑Lance index exhibits a monotonic increase with the number of clusters yet still yields a high Pearson correlation (0.93), masking its poor discriminative power.

The dataset collection dramatically expands the experimental landscape. In addition to the 972 datasets used in the prior benchmark, the authors generate 15 205 synthetic datasets covering a broad spectrum of dimensionality, cluster counts, densities, noise levels, and cluster shapes. This diversity enables a robust assessment of index behavior across realistic and pathological scenarios. Eight clustering algorithms—K‑Means, Spectral Clustering, Agglomerative (Single Linkage), HDBSCAN*, Trimmed K‑Means, Fuzzy C‑Means, EM‑GMM, and DBSCAN—produce candidate partitions, ensuring that algorithmic biases are also examined.

Results show that classic indexes such as Silhouette, Dunn, and Calinski‑Harabasz perform well when the number of clusters matches the ground truth but deteriorate sharply when k is under‑ or over‑estimated. Modern density‑based and information‑theoretic indexes (e.g., DBCV, Adjusted Mutual Information based measures) maintain higher accuracy (≈ 85 % correct selection) and more stable correlations across all scenarios. Some indexes, notably Ratkowski‑Lance and certain versions of Dunn, display systematic monotonic trends with k, leading to misleadingly high Pearson scores despite poor ranking ability.

The authors also conduct statistical validation using bootstrapped confidence intervals, Friedman tests, and Nemenyi post‑hoc analysis, confirming that the observed differences are statistically significant. They argue that a single correlation value is insufficient to capture the nuanced behavior of internal validity measures; a combination of accuracy, multiple correlation metrics, and visual inspection is necessary for a reliable assessment.

In conclusion, the paper delivers three major contributions: (1) a diversified, bias‑aware evaluation protocol that mitigates the “single‑optimal‑partition” pitfall; (2) a thorough investigation of linear and non‑linear relationships between internal and external indices; and (3) the release of a massive, publicly available benchmark suite. The findings suggest that practitioners should favor newer density‑ or information‑theoretic indexes for robust clustering validation, while remaining cautious of traditional measures that may be overly sensitive to the number of clusters. Future work is outlined in the direction of meta‑learning for automatic index selection, domain‑specific index design, and integration of the benchmark into real‑time clustering pipelines.