VillageNet: Graph-based, Easily-interpretable, Unsupervised Clustering for Broad Biomedical Applications

Clustering large high-dimensional datasets with diverse variable is essential for extracting high-level latent information from these datasets. Here, we developed an unsupervised clustering algorithm, we call “Village-Net”. Village-Net is specifically designed to effectively cluster high-dimension data without priori knowledge on the number of existing clusters. The algorithm operates in two phases: first, utilizing K-Means clustering, it divides the dataset into distinct subsets we refer to as “villages”. Next, a weighted network is created, with each node representing a village, capturing their proximity relationships. To achieve optimal clustering, we process this network using a community detection algorithm called Walk-likelihood Community Finder (WLCF), a community detection algorithm developed by one of our team members. A salient feature of Village-Net Clustering is its ability to autonomously determine an optimal number of clusters for further analysis based on inherent characteristics of the data. We present extensive benchmarking on extant real-world datasets with known ground-truth labels to showcase its competitive performance, particularly in terms of the normalized mutual information (NMI) score, when compared to other state-of-the-art methods. The algorithm is computationally efficient, boasting a time complexity of O(Nkd), where N signifies the number of instances, k represents the number of villages and d represents the dimension of the dataset, which makes it well suited for effectively handling large-scale datasets.

💡 Research Summary

VillageNet is a novel unsupervised clustering framework designed to handle high‑dimensional biomedical data without requiring prior knowledge of the number of clusters. The method proceeds in two main stages. First, the entire dataset is deliberately over‑clustered using K‑Means into a large number (ν) of small, locally linear partitions called “villages”. Each village corresponds to a Voronoi cell around a K‑Means centroid, ensuring that within a village the data geometry can be approximated as linear even when the global structure is highly non‑linear.

In the second stage, the relationships between villages are quantified by defining an “exterior” set for each village: the η data points that lie closest to the boundary between the village and any neighboring village. The weight of an edge between two villages U and V is the sum of the number of U‑exterior points assigned to V and the number of V‑exterior points assigned to U. This weighted graph, termed the “village network”, captures the density of points along inter‑village boundaries; larger weights indicate strong connectivity, while smaller weights signal weak links.

The village network is then partitioned using the Walk‑Likelihood Community Finder (WLCF), a random‑walk‑based community detection algorithm that combines maximum‑likelihood estimates of node visit frequencies with modularity optimization. Crucially, WLCF does not require the number of communities as an input; the optimal number of communities emerges naturally from the random‑walk dynamics. Final clusters are obtained by merging all data points belonging to villages that fall within the same community, guaranteeing non‑overlapping clusters.

The authors provide a detailed computational complexity analysis. The dominant cost is the initial K‑Means over‑clustering, which scales as O(N·ν·d) where N is the number of instances, ν the number of villages, and d the dimensionality. Subsequent steps—exterior identification (O(N·ν)), graph construction (effectively O(N·ν²) but accelerated by optimized matrix operations), and WLCF (empirically O(ν¹·⁵))—are all sub‑dominant when ν ≪ N. Consequently, VillageNet exhibits near‑linear scaling with dataset size, making it suitable for large‑scale biomedical applications.

Hyperparameters ν (village count) and η (exterior size) control the granularity of the representation. Larger ν yields finer villages and typically higher clustering accuracy at the expense of longer runtimes; η determines how many nearest‑boundary points are considered when building the graph, balancing local detail against graph sparsity. The paper includes a systematic study on the digits dataset showing how NMI varies with these parameters, and recommends selecting the smallest ν that still captures the underlying structure while keeping η moderate to avoid over‑smoothing.

Performance is benchmarked on a suite of non‑biomedical datasets with known ground‑truth labels (e.g., MNIST, digits, 20 Newsgroups) and on four heterogeneous biomedical datasets: flow cytometry, tissue imaging, single‑cell RNA‑seq, and image‑derived cellular profiling. Across all tests, VillageNet achieves normalized mutual information (NMI) and adjusted Rand index (ARI) scores that are competitive with or superior to state‑of‑the‑art methods such as Louvain, Leiden, Phenograph, DBSCAN, OPTICS, and various kernel‑based K‑Means variants. Notably, VillageNet excels when clusters are non‑linearly separable, and it does so without any dimensionality reduction, preserving the full information content of high‑dimensional measurements.

The manuscript also discusses limitations. The choice of ν and η can be data‑dependent, and the initial K‑Means seeding may affect village formation. Moreover, WLCF is a proprietary algorithm developed by one of the authors, and its public implementation is not yet widely available, which could hinder reproducibility. Future work is suggested to explore automatic hyper‑parameter tuning, alternative initial partitioning schemes (e.g., Gaussian mixture models), and broader validation of WLCF on diverse graph structures.

In summary, VillageNet offers a conceptually simple yet powerful pipeline—over‑cluster → construct weighted inter‑village graph → random‑walk community detection—that automatically discovers the appropriate number of clusters and scales linearly with data size. Its ability to handle heterogeneous, high‑dimensional biomedical data without prior assumptions makes it a promising tool for precision‑medicine research, where uncovering latent patient subgroups or cellular phenotypes is often the first step toward targeted therapeutic strategies.