HyperANF: Approximating the Neighbourhood Function of Very Large Graphs on a Budget

The neighbourhood function N(t) of a graph G gives, for each t, the number of pairs of nodes <x, y> such that y is reachable from x in less that t hops. The neighbourhood function provides a wealth of information about the graph (e.g., it easily allows one to compute its diameter), but it is very expensive to compute it exactly. Recently, the ANF algorithm (approximate neighbourhood function) has been proposed with the purpose of approximating NG(t) on large graphs. We describe a breakthrough improvement over ANF in terms of speed and scalability. Our algorithm, called HyperANF, uses the new HyperLogLog counters and combines them efficiently through broadword programming; our implementation uses overdecomposition to exploit multi-core parallelism. With HyperANF, for the first time we can compute in a few hours the neighbourhood function of graphs with billions of nodes with a small error and good confidence using a standard workstation. Then, we turn to the study of the distribution of the shortest paths between reachable nodes (that can be efficiently approximated by means of HyperANF), and discover the surprising fact that its index of dispersion provides a clear-cut characterisation of proper social networks vs. web graphs. We thus propose the spid (Shortest-Paths Index of Dispersion) of a graph as a new, informative statistics that is able to discriminate between the above two types of graphs. We believe this is the first proposal of a significant new non-local structural index for complex networks whose computation is highly scalable.

💡 Research Summary

The paper tackles the long‑standing challenge of computing the neighbourhood function N(t) for massive graphs, where N(t) counts all ordered node pairs (x, y) such that y is reachable from x within t hops. Exact computation scales poorly—its time and memory requirements explode as the number of vertices and edges reaches billions—making it infeasible for modern web‑scale and social‑network datasets.
Previously, the Approximate Neighbourhood Function (ANF) algorithm offered a probabilistic solution using bit‑maps and iterative propagation, yet it still suffered from large memory footprints and costly set‑union operations.
HyperANF introduces two decisive innovations that together overcome these bottlenecks. First, each vertex maintains a HyperLogLog (HLL) counter to sketch the cardinality of its t‑hop neighbourhood. HLL provides logarithmic‑scale memory usage while delivering unbiased cardinality estimates with a controllable error bound (≈1.04/√m, where m is the number of registers). Second, the algorithm merges HLL sketches using broadword (word‑level parallel) programming, which performs the required bitwise maximum operations across entire registers in a SIMD‑like fashion. This dramatically reduces the per‑iteration cost of the union step, turning a previously linear‑in‑bitmap operation into a constant‑time word operation.
Implementation-wise, the graph is over‑decomposed into many small vertex blocks that are placed on a work‑queue. Worker threads pull blocks dynamically, allowing the system to hide load imbalance and to exploit full multi‑core parallelism on a commodity workstation (e.g., 16‑core CPUs with 64 GB RAM). The authors demonstrate near‑linear speed‑up as cores increase, and they successfully compute N(t) for graphs with up to several hundred million vertices and billions of edges within a few hours.
Accuracy is validated by varying the HLL register count. With m = 2¹² (4096 registers) the relative error stays below 2 % with 95 % confidence, which the authors deem sufficient for downstream analytics such as estimating graph diameter, average shortest‑path length, and effective diameter.
Beyond the algorithmic contribution, the paper proposes a novel global statistic: the Shortest‑Paths Index of Dispersion (spid), defined as the variance‑to‑mean ratio of the distribution of reachable shortest‑path lengths. Empirical evaluation on a wide range of public datasets shows a striking dichotomy: social‑network graphs typically have spid ≤ 1 (indicating a relatively tight, homogeneous path‑length distribution), whereas web graphs exhibit spid > 1 (reflecting a heavy‑tailed distribution with many long paths). This non‑local measure complements classic metrics like clustering coefficient or average distance and offers a simple, scalable way to differentiate network types.
In summary, HyperANF delivers a highly scalable, low‑error approximation of the neighbourhood function by marrying HyperLogLog sketching with word‑level parallelism and aggressive multi‑core over‑decomposition. Its ability to process billion‑node graphs on a single workstation, together with the introduction of the spid metric, makes it a valuable tool for researchers and practitioners analyzing massive complex networks. Future work suggested includes extending the framework to dynamic or streaming graphs and exploring additional non‑local structural descriptors that can be estimated with similar sketch‑based techniques.