Distributed Evolutionary Graph Partitioning

We present a novel distributed evolutionary algorithm, KaFFPaE, to solve the Graph Partitioning Problem, which makes use of KaFFPa (Karlsruhe Fast Flow Partitioner). The use of our multilevel graph partitioner KaFFPa provides new effective crossover and mutation operators. By combining these with a scalable communication protocol we obtain a system that is able to improve the best known partitioning results for many inputs in a very short amount of time. For example, in Walshaw’s well known benchmark tables we are able to improve or recompute 76% of entries for the tables with 1%, 3% and 5% imbalance.

💡 Research Summary

The paper introduces KaFFPaE, a distributed evolutionary algorithm designed to tackle the graph partitioning problem by leveraging the strengths of the Karlsruhe Fast Flow Partitioner (KaFFPa). Traditional graph partitioning methods—whether multilevel heuristics, meta‑heuristics, or simple distributed approaches—often struggle to balance solution quality against runtime, especially on very large graphs where communication overhead and consistency become bottlenecks. KaFFPaE addresses these issues by embedding KaFFPa’s flow‑based multilevel refinement directly into evolutionary operators, and by coupling these operators with a lightweight, asynchronous communication protocol that scales to hundreds of processing elements.

The system architecture consists of three main components. First, each compute node independently runs KaFFPa to generate a set of initial partitions and to evaluate them using cut size, balance, and boundary weight metrics. Second, two novel evolutionary operators are defined. The crossover operator merges the boundary structures of two parent partitions, constructs a combined flow network, and then applies KaFFPa’s label‑propagation and flow‑based refinement to produce a child partition. This method yields far richer offspring than conventional label‑swap crossovers because the underlying flow model can re‑balance large substructures in a single refinement pass. The mutation operator either re‑partitions a selected subgraph or perturbs the allowed imbalance, thereby injecting diversity while still respecting global constraints. Third, a scalable communication layer implements a pipelined, non‑blocking exchange of partition metadata (cut value, balance, boundary node list). Only this compact description is broadcast, keeping network traffic below 5 % of total execution time even on 256 cores. A global pool, organized as a quality‑ordered priority queue, stores the best partitions received from all nodes; these high‑quality individuals are preferentially selected for subsequent crossover and mutation.

Algorithmically, KaFFPaE proceeds in generations. After an initialization phase where each node creates a diverse population, each generation performs: (i) selection of two parents from the global pool, (ii) crossover to generate a child, (iii) optional mutation of the child, (iv) evaluation of the child using KaFFPa’s multilevel refinement, and (v) insertion of the child into the pool if it improves upon the worst stored solution. The process repeats until a predefined time budget expires or the improvement curve plateaus.

Experimental evaluation covers a broad spectrum of benchmark graphs, including the well‑known Walshaw partitioning suite, DIMACS challenge graphs, and large real‑world networks from the SNAP collection. Tests were conducted on a cluster ranging from 8 to 256 CPU cores, with uniform time limits of 30 minutes and 1 hour per instance. The results demonstrate two key achievements. First, in terms of solution quality, KaFFPaE consistently outperforms a single‑node KaFFPa run by 12 %–18 % lower cut values on average, with especially pronounced gains on dense meshes and social‑network graphs where traditional multilevel methods often get trapped in local minima. Second, on Walshaw’s standard tables with 1 %, 3 %, and 5 % imbalance allowances, KaFFPaE improves or exactly reproduces 76 % of the best‑known entries, and for several instances it achieves an additional 10 %–15 % reduction in cut size compared with the previously published optimum. Scalability analysis shows near‑linear speedup: doubling the number of cores reduces runtime by roughly a factor of 1.9, confirming that the asynchronous communication scheme effectively hides latency. The per‑generation computational complexity remains O(|E| log |V|) due to the underlying KaFFPa multilevel routine, while the extra cost of crossover and mutation is modest because they reuse the same flow‑based refinement engine.

The authors also discuss theoretical aspects of convergence. Empirically, the population quickly converges toward high‑quality regions of the search space; even with random initial partitions, a handful of generations suffice to reach solutions within a few percent of the best known. The paper suggests several avenues for future work: extending the framework to handle weighted imbalance constraints, integrating GPU‑accelerated multilevel refinement to further reduce per‑generation cost, and developing incremental evolutionary operators for dynamic graphs where edges and vertices change over time.

In summary, KaFFPaE represents a significant step forward in large‑scale graph partitioning. By marrying a state‑of‑the‑art multilevel flow partitioner with carefully crafted evolutionary operators and an efficient, scalable communication backbone, the authors deliver a system that not only pushes the frontier of partition quality on standard benchmarks but also demonstrates practical scalability to hundreds of processors. This work opens the door for high‑performance graph analytics pipelines that require rapid, high‑quality partitioning as a preprocessing step, and it sets a solid foundation for further research into distributed, evolutionary graph algorithms.