Scaling Distributed All-Pairs Algorithms: Manage Computation and Limit Data Replication with Quorums

In this paper we propose and prove that cyclic quorum sets can efficiently manage all-pairs computations and data replication. The quorums are O(N/sqrt(P)) in size, up to 50% smaller than the dual N/sqrt(P) array implementations, and significantly smaller than solutions requiring all data. Implementation evaluation demonstrated scalability on real datasets with a 7x speed up on 8 nodes with 1/3rd the memory usage per process. The all-pairs problem requires all data elements to be paired with all other data elements. These all-pair problems occur in many science fields, which has led to their continued interest. Additionally, as datasets grow in size, new methods like these that can reduce memory footprints and distribute work equally across compute nodes will be demanded.

💡 Research Summary

The paper addresses the classic all‑pairs problem, where every element of a data set must be paired with every other element, a task that appears in many scientific domains such as genomics, particle simulations, and graph analytics. Traditional distributed solutions either replicate the entire data set on each node (resulting in O(N) memory per process) or partition the data into N/√P blocks arranged in a two‑dimensional grid. Both approaches suffer from excessive memory consumption and communication overhead, especially as the number of processes P grows.

To overcome these limitations, the authors propose using cyclic quorum sets, a combinatorial construct originally studied for fault‑tolerant replication. In a system with P processes, the data set of size N is divided into √P “slices”. Each process is assigned a quorum consisting of √P consecutive slices, but the starting slice for each process is shifted cyclically around the ring of slices. Formally, the quorum Qp for process p contains all data indices whose slice index lies in the interval