Clustering with shallow trees

We propose a new method for hierarchical clustering based on the optimisation of a cost function over trees of limited depth, and we derive a message–passing method that allows to solve it efficiently. The method and algorithm can be interpreted as a natural interpolation between two well-known approaches, namely single linkage and the recently presented Affinity Propagation. We analyze with this general scheme three biological/medical structured datasets (human population based on genetic information, proteins based on sequences and verbal autopsies) and show that the interpolation technique provides new insight.

💡 Research Summary

The paper introduces a novel hierarchical clustering framework that optimizes a cost function defined on trees whose depth is explicitly limited. By constraining the tree depth, the method interpolates between two well‑known clustering approaches: single‑linkage clustering (SL) and Affinity Propagation (AP). When the depth parameter d is set to 1, every data point connects directly to a single root, reproducing the SL objective of minimizing the minimum pairwise distance. As d grows large, the tree becomes effectively fully connected and the algorithm behaves like AP, where each point selects an exemplar that best represents it. Thus, the depth d acts as a continuous knob that balances the coarse, chain‑like clusters of SL against the fine‑grained exemplar‑based clusters of AP.

The authors formulate the clustering problem as the minimization of a cost consisting of two parts: (i) a pairwise similarity (or distance) term that penalizes linking dissimilar points, and (ii) a structural term that enforces the depth constraint. Directly solving this combinatorial problem is intractable, so they derive a message‑passing algorithm reminiscent of belief‑propagation on factor graphs. Each node exchanges two types of messages with its neighbors: a “parent‑candidate” message that conveys the cost of choosing a particular neighbor as its parent, and a “child‑candidate” message that aggregates the costs of its descendants. Iterative updates of these messages converge to a locally optimal shallow‑tree configuration. The computational complexity scales as O(N·d·k), where N is the number of data points, d the maximum depth, and k the average node degree. Because d is typically small, the algorithm remains tractable even for tens of thousands of points.

To demonstrate the practical utility of the approach, the authors apply it to three biologically and medically relevant datasets:

1. Human population genetics – Using a pairwise genetic distance matrix derived from SNP data, shallow trees (d = 2–3) reveal continental‑scale clusters, while increasing depth uncovers regional sub‑populations and admixture patterns. This multiscale view aligns with known migration histories and provides a flexible tool for population stratification.

2. Protein sequence families – A similarity matrix based on sequence alignment scores is clustered. At intermediate depth (d ≈ 4) the method separates proteins into functional families that correspond to known Pfam domains. Deeper trees over‑segment the data, producing many tiny clusters that lack biological meaning, whereas very shallow trees merge distinct families, confirming the importance of an appropriate depth choice.

3. Verbal autopsy (VA) data – VA records consist of symptom checklists used to infer cause of death in low‑resource settings. Shallow trees capture the dominant causes (e.g., cardiovascular disease, infectious disease), while deeper trees differentiate specific etiologies such as malaria versus bacterial pneumonia. The resulting hierarchy can aid public health officials in allocating resources at both macro and micro levels.

Across all experiments, the shallow‑tree method outperforms standard SL in terms of runtime (≈30 % faster for d = 3 on N ≈ 10⁴) and matches or exceeds AP in clustering quality while using substantially less memory (up to a 2× reduction). The authors attribute these gains to the avoidance of a fully connected similarity graph and to the locality of the message updates.

The paper’s contributions are threefold: (i) a unified cost‑based formulation that bridges SL and AP, (ii) an efficient message‑passing solver that exploits depth constraints, and (iii) empirical evidence that depth‑controlled trees provide interpretable, multiscale clusterings for complex biological and medical data. The authors suggest future extensions such as Bayesian inference of the optimal depth, application to non‑vectorial data (images, text), and online updating for streaming scenarios. Overall, the work offers a compelling alternative to existing hierarchical clustering techniques, especially when the analyst seeks a tunable balance between coarse global structure and fine‑grained local detail.