Community detection for interaction networks

In many applications, it is common practice to obtain a network from interaction counts by thresholding each pairwise count at a prescribed value. Our analysis calls attention to the dependence of certain methods, notably Newman–Girvan modularity, on the choice of threshold. Essentially, the threshold either separates the network into clusters automatically, making the algorithm’s job trivial, or erases all structure in the data, rendering clustering impossible. By fitting the original interaction counts as given, we show that minor modifications to classical statistical methods outperform the prevailing approaches for community detection from interaction datasets. We also introduce a new hidden Markov model for inferring community structures that vary over time. We demonstrate each of these features on three real datasets: the karate club dataset, voting data from the U.S.\ Senate (2001–2003), and temporal voting data for the U.S. Supreme Court (1990–2004).

💡 Research Summary

The paper addresses a fundamental problem in community detection for interaction networks: the common practice of converting rich interaction count data into a binary adjacency matrix by applying a threshold. The authors demonstrate that this preprocessing step can dramatically affect the performance of popular algorithms, especially Newman‑Girvan modularity, which may either trivially separate the network into clusters when the threshold is too low or completely erase any structure when the threshold is too high. To overcome this fragility, the authors propose to model the original interaction counts directly, thereby avoiding any arbitrary discretisation.

Three methodological contributions are presented. First, a two‑parameter Poisson stochastic block model (Poisson SBM) is introduced. It assumes that within‑cluster and between‑cluster interaction counts follow Poisson distributions with different means. Despite its simplicity—only two parameters per block—the model perfectly recovers the known two‑party split in Zachary’s karate club data, outperforming the degree‑corrected stochastic block model (DC‑SBM) which requires many more parameters and misclassifies one member.

Second, the paper adapts classical statistical clustering techniques (e.g., EM for mixture of Poisson distributions, BIC‑based model selection) to the interaction‑count setting. By fitting these models directly to the count matrix, the authors obtain stable, interpretable partitions that are robust to any choice of threshold. Empirical results on the U.S. Senate voting data (100 senators, thousands of bill‑by‑bill agreements) and the U.S. Supreme Court voting data (nine justices over fifteen terms) show higher log‑likelihoods and lower variation of information compared with network‑based methods.

Third, to capture temporal evolution of communities, the authors develop a hidden Markov model (HMM) on partitions, also known as a partition‑valued Markov chain. Each time slice’s interaction matrix is treated as an observation generated from a latent partition that evolves according to a Markov transition matrix. Using Bayesian inference and MCMC sampling, the model detects shifts in the ideological alignment of the Supreme Court between 1990 and 2004, revealing subtle regime changes that static models miss.

Across all three real‑world examples, the proposed approaches consistently outperform threshold‑based network methods. The paper includes detailed sensitivity analyses (e.g., Figure 6.2, Table 1) illustrating how Newman‑Girvan modularity’s output varies wildly with different cut‑offs, whereas the Poisson SBM and the HMM produce virtually identical partitions regardless of any threshold. The authors argue that ignoring the sampling mechanism—i.e., the process of turning counts into edges—can lead to misleading inferences, echoing earlier work on network sampling bias.

In summary, the contributions are: (1) a principled framework for community detection that operates directly on interaction count data, eliminating the need for arbitrary thresholds; (2) evidence that a simple Poisson block model can replace more complex degree‑corrected models without loss of accuracy; (3) a novel hidden Markov model for time‑varying community structures, demonstrated on Supreme Court voting; and (4) a systematic demonstration of the pitfalls of thresholding, reinforcing the statistical principle that the simplest adequate model is often the best. This work offers a clear, statistically sound alternative for researchers dealing with interaction‑rich datasets in social, political, and biological sciences.