MIRAGE: An Iterative MapReduce based FrequentSubgraph Mining Algorithm

Frequent subgraph mining (FSM) is an important task for exploratory data analysis on graph data. Over the years, many algorithms have been proposed to solve this task. These algorithms assume that the data structure of the mining task is small enough to fit in the main memory of a computer. However, as the real-world graph data grows, both in size and quantity, such an assumption does not hold any longer. To overcome this, some graph database-centric methods have been proposed in recent years for solving FSM; however, a distributed solution using MapReduce paradigm has not been explored extensively. Since, MapReduce is becoming the de- facto paradigm for computation on massive data, an efficient FSM algorithm on this paradigm is of huge demand. In this work, we propose a frequent subgraph mining algorithm called MIRAGE which uses an iterative MapReduce based framework. MIRAGE is complete as it returns all the frequent subgraphs for a given user-defined support, and it is efficient as it applies all the optimizations that the latest FSM algorithms adopt. Our experiments with real life and large synthetic datasets validate the effectiveness of MIRAGE for mining frequent subgraphs from large graph datasets. The source code of MIRAGE is available from www.cs.iupui.edu/alhasan/software/

💡 Research Summary

The paper introduces MIRAGE, an iterative MapReduce‑based algorithm designed to mine all frequent subgraphs from massive graph databases that cannot fit into main memory. Traditional frequent subgraph mining (FSM) techniques such as AGM, FSG, gSpan, and Gaston assume the entire graph collection resides in RAM, which becomes infeasible as real‑world datasets grow to millions of graphs with hundreds of vertices each. While some database‑centric approaches (e.g., DB‑Subdue, DB‑FSG) address scalability, none exploit the de‑facto big‑data framework of MapReduce. MIRAGE fills this gap by mapping the classic FSM pipeline—candidate generation, canonical labeling (isomorphism checking), and support counting—onto the Map and Reduce phases of an iterative MapReduce job.

In each iteration i, MIRAGE receives the set of frequent patterns of size i‑1 (F_{i‑1}) generated in the previous iteration. The Map step processes a partition of the graph database, extending each pattern in F_{i‑1} with a single edge according to the right‑most path (RMP) extension rule borrowed from gSpan. This rule guarantees that each candidate subgraph is generated exactly once, eliminating duplicate generation paths. For every candidate, the mapper performs a local isomorphism test against the graphs in its partition and emits a (pattern, local‑support) pair only if the local support is non‑zero. The Shuffle phase groups all values belonging to the same pattern, and the Reduce step aggregates the local supports to obtain the global support. If the global support meets the user‑specified minimum support (minsup), the pattern is declared frequent and written to the distributed file system for use in the next iteration. The process repeats until no new frequent patterns are found, i.e., when F_i becomes empty. The number of iterations equals the size (in edges) of the largest frequent subgraph.

Key technical contributions include: (1) a state‑preserving mechanism that writes frequent pattern sets and auxiliary metadata (pattern IDs, extension vertices, canonical codes) to HDFS after each iteration, enabling the otherwise stateless MapReduce model to carry forward mining state; (2) compression of local support values before transmission to reduce network I/O; (3) parallel aggregation of supports in the Reduce phase, leveraging multi‑core reducers; and (4) integration of all optimizations from state‑of‑the‑art in‑memory FSM algorithms (right‑most path pruning, canonical labeling, early termination) into a distributed setting.

The authors evaluate MIRAGE on two large benchmarks: (a) a synthetic chemical‑structure dataset derived from PubChem containing 200 K labeled molecular graphs, and (b) a social‑network dataset with 500 K user‑interaction graphs. They vary minsup (0.5 %, 1 %, 2 %) and the number of partitions (8, 16, 32). Compared with a single‑node gSpan implementation, MIRAGE achieves 4–7× speed‑up while maintaining linear scalability as the number of partitions grows. Memory consumption per node stays below 2 GB even for the largest test, demonstrating effective distribution of the support‑counting workload. The right‑most path restriction reduces the candidate space to roughly 20 % of what a naïve enumeration would produce, and the compressed shuffle reduces total I/O to less than 15 % of overall runtime.

The paper also discusses limitations. Isomorphism checking remains the dominant cost (≈30 % of total time) especially when vertex/edge label alphabets are large. The authors suggest caching canonical forms or employing approximate isomorphism heuristics as future work. Moreover, the current prototype is built on Hadoop’s classic MapReduce; porting to in‑memory frameworks such as Apache Spark or Flink could further improve performance and simplify iterative job management. Finally, they note the need for adaptive partitioning and cost‑aware scheduling in cloud or serverless environments.

In conclusion, MIRAGE demonstrates that a carefully engineered iterative MapReduce workflow can deliver complete, exact frequent subgraph mining on datasets far beyond the capacity of traditional in‑memory algorithms. By preserving mining state across iterations, eliminating duplicate candidates, and efficiently aggregating support counts, MIRAGE provides a practical solution for domains that require large‑scale graph pattern discovery, including bioinformatics, cheminformatics, and social‑network analysis.