Improving the performance of algorithms to find communities in networks

Many algorithms to detect communities in networks typically work without any information on the cluster structure to be found, as one has no a priori knowledge of it, in general. Not surprisingly, knowing some features of the unknown partition could help its identification, yielding an improvement of the performance of the method. Here we show that, if the number of clusters were known beforehand, standard methods, like modularity optimization, would considerably gain in accuracy, mitigating the severe resolution bias that undermines the reliability of the results of the original unconstrained version. The number of clusters can be inferred from the spectra of the recently introduced non-backtracking and flow matrices, even in benchmark graphs with realistic community structure. The limit of such two-step procedure is the overhead of the computation of the spectra.

💡 Research Summary

The paper investigates how prior knowledge of the number of communities (clusters) in a network can dramatically improve the performance of community‑detection algorithms, especially modularity optimization, which suffers from a well‑known resolution limit. The authors propose a two‑step procedure: first, infer the number of clusters q from the spectra of two recently introduced graph matrices—the non‑backtracking matrix B and the flow matrix F; second, run a standard community‑detection method (e.g., Newman‑Girvan modularity maximization or the Absolute Potts Model) while constraining the search to partitions with exactly q clusters.

The theoretical background is built on the planted partition model, where N nodes are divided into q equally sized groups with intra‑group connection probability pin and inter‑group probability pout. Detectability in sparse graphs requires pin > pout + Δ, where Δ depends on the model parameters. The authors show that, for q = 2 and various cluster sizes (n = 50–1000) with fixed internal degree μin = 10, the fraction of correctly classified nodes drops sharply as the external degree μout approaches the detectability limit. When q is supplied to the algorithm (denoted Mod+q and APM+q), both modularity maximization and the Absolute Potts Model achieve performance close to the theoretical optimum, essentially eliminating the resolution limit. In contrast, unconstrained runs (Mod, APM) perform substantially worse, especially near the limit.

To obtain q, the paper leverages the spectral properties of B and F. Both are 2m × 2m sparse matrices (m = number of edges) that encode directed edge adjacency (B) and a degree‑normalized version (F). Most eigenvalues lie inside a circle centered at the origin; eigenvalues that fall outside this bulk correspond to community structure. The number of out‑of‑bulk eigenvalues provides an estimate of q. Experiments on the planted partition model show that this estimate is highly accurate, improving with larger graph sizes. The authors also compare against community‑detection methods that infer q internally (e.g., OSLOM, Infomap). While Infomap performs relatively well on the LFR benchmark, it still under‑estimates q for certain parameter regimes, whereas the spectral methods remain more reliable.

The authors extend the analysis to the LFR benchmark, which incorporates realistic heterogeneous degree distributions and community size heterogeneity. They test networks of 1 000 and 5 000 nodes, with small (10–50 nodes) and big (20–100 nodes) communities, average degree 20, and mixing parameter μ ranging from 0 (well‑separated) to 1 (highly mixed). The spectral methods (B and F) continue to predict q accurately for low μ, though accuracy degrades as μ increases. Infomap again yields the best q estimates among the detection algorithms, but the spectral approach remains competitive and does not suffer from the resolution limit that hampers modularity.

A major practical limitation identified is the computational cost of obtaining the full spectra of B or F, which scales roughly as O(m √m) and becomes prohibitive for networks with more than about one million edges. The authors suggest that developing fast approximate spectral techniques would be essential for scaling the two‑step method to very large graphs.

In summary, the study demonstrates that (1) knowing the number of communities a priori allows constrained modularity‑based methods to reach near‑optimal detection performance, effectively bypassing the resolution limit; (2) the number of communities can be reliably inferred from the out‑of‑bulk eigenvalues of the non‑backtracking or flow matrices, even on realistic benchmark graphs; and (3) the primary bottleneck is the spectral computation, motivating future work on efficient approximations. This two‑step framework offers a promising route to more accurate community detection when modest prior information can be extracted from the network’s spectral signatures.