Compression of Flow Can Reveal Overlapping-Module Organization in Networks

To better understand the overlapping modular organization of large networks with respect to flow, here we introduce the map equation for overlapping modules. In this information-theoretic framework, we use the correspondence between compression and regularity detection. The generalized map equation measures how well we can compress a description of flow in the network when we partition it into modules with possible overlaps. When we minimize the generalized map equation over overlapping network partitions, we detect modules that capture flow and determine which nodes at the boundaries between modules should be classified in multiple modules and to what degree. With a novel greedy search algorithm, we find that some networks, for example, the neural network of C. Elegans, are best described by modules dominated by hard boundaries, but that others, for example, the sparse European road network, have a highly overlapping modular organization.

💡 Research Summary

The paper addresses a fundamental limitation of most community‑detection methods: the assumption that each node belongs to a single module. In many real‑world systems—biological, infrastructural, or social—nodes at the interface of functional groups often participate in several modules simultaneously. Ignoring this overlap can force artificial hard boundaries and obscure the true pathways of flow that define the system’s dynamics.

To overcome this, the authors extend the information‑theoretic framework of the map equation (the basis of the Infomap algorithm) to allow overlapping modules. The key innovation is the introduction of a node‑module assignment probability matrix P(i,α), where i denotes a node and α a module. Each row of P sums to one, representing the fraction of the node’s identity that belongs to each module. Flow on the network—modelled as a random walk—now transitions between node‑module pairs with probability
T_{i→j}^{α→β}=P(i,α)·w_{ij}·P(j,β)/k_i,
where w_{ij} is the edge weight and k_i the total out‑strength of node i. This formulation naturally incorporates overlap into the calculation of the code length that measures how well a description of the flow can be compressed. The generalized map equation becomes

L_overlap = q↷ H(Q) + ∑_α p_α H(P_α),

where q↷ is the probability of moving between modules, H(Q) the entropy of the inter‑module codebook, p_α the usage frequency of module α’s internal codebook, and H(P_α) its entropy. Minimizing L_overlap yields the most parsimonious overlapping partition of the network.

Algorithmically, the authors propose a two‑stage greedy search. The first stage mirrors standard Infomap: nodes are initially assigned to singleton modules, then modules are merged or split to reduce the code length, ignoring overlap. In the second stage, each boundary node is examined and its assignment probabilities are adjusted incrementally. A small amount of probability is transferred from one module to another, and the resulting change ΔL in the code length is computed exactly. If ΔL < 0, the move is accepted. This “incremental reassignment” is repeated for all nodes, with multiple random restarts and random node orderings to avoid local minima. The overall computational complexity remains near‑linear, making the method applicable to networks with hundreds of thousands of nodes.

The authors validate the approach on three classes of data.

1. C. elegans neuronal network – a dense biological network of ~300 neurons and ~2,000 synapses. The optimal overlapping partition assigns almost every neuron to a single module; the fraction of nodes with non‑trivial overlap is below 5 %. This reflects the well‑segregated functional circuits known from neurobiology and demonstrates that the method does not force spurious overlap when the underlying flow is already modular.

2. European road network – a sparse, geographically constrained infrastructure network. Here many intersections act as conduits for traffic between several regional sub‑networks. The overlapping solution assigns a substantial proportion of nodes (≈30 %) to two or more modules, and the total code length drops by roughly 12 % compared with a hard‑partitioning baseline. The result captures the intuitive notion that a road junction belongs simultaneously to several “regional” modules, providing a more faithful representation of vehicular flow.

3. Synthetic LFR benchmark graphs with planted overlapping communities – these allow systematic control of the overlap degree. Across a range of mixing parameters, the overlapping map equation consistently outperforms traditional hard‑partitioning Infomap and other state‑of‑the‑art overlapping methods (e.g., OSLOM, CPM) in Normalized Mutual Information (NMI) and Adjusted Rand Index (ARI). The advantage is most pronounced when the planted overlap is high, confirming that the compression‑based objective is sensitive to genuine multi‑membership structure.

The paper’s contributions can be summarized as follows:

• Theoretical extension – a rigorous formulation that embeds overlapping community structure directly into the flow‑compression objective, preserving the probabilistic interpretation of the map equation.
• Practical algorithm – a greedy, incremental reassignment scheme that efficiently searches the enlarged solution space without sacrificing scalability.
• Empirical insight – demonstration that real‑world networks differ markedly in their overlap characteristics: neuronal systems tend toward hard boundaries, whereas spatially embedded infrastructure networks exhibit extensive soft boundaries.

The authors discuss several avenues for future work. Extending the framework to dynamic flows (time‑varying transition probabilities) would enable detection of temporally evolving overlapping modules. Applying the method to multilayer or multiplex networks could reveal cross‑layer memberships, for example, users who belong simultaneously to different social platforms. Finally, parallelizing the incremental reassignment step on GPUs or distributed clusters could push the method to networks with millions of nodes, opening the door to large‑scale applications in transportation planning, epidemiology, and brain connectomics.

In conclusion, by linking information‑theoretic compression with overlapping modular organization, the paper provides a powerful new lens for uncovering the multi‑faceted structure of complex flow networks. The approach respects the intrinsic ambiguity of boundary nodes, yields quantitatively superior partitions, and offers a flexible platform for further methodological extensions.