Interleaved computation for persistent homology

We describe an approach to bounded-memory computation of persistent homology and betti barcodes, in which a computational state is maintained with updates introducing new edges to the underlying neighbourhood graph and percolating the resulting changes into the simplex stream feeding the persistence algorithm. We further discuss the memory consumption and resulting speed and complexity behaviours of the resulting algorithm.

💡 Research Summary

The paper introduces an interleaved computation framework for persistent homology that dramatically reduces memory consumption while allowing continuous, incremental processing of Vietoris‑Rips complexes. Traditional pipelines first generate the entire Vietoris‑Rips complex (or a fixed‑dimensional skeleton) and then feed the complete simplex stream into a persistence algorithm. This approach requires exponential space in the number of points and the chosen simplex dimension, making it infeasible for large data sets and for exploratory workflows that need repeated parameter tuning.

The authors propose to maintain a dynamic list of maximal cliques of the underlying neighbourhood graph. When a new edge (e = (s,t)) arrives, the algorithm identifies the maximal cliques containing (s) and (t) (denoted (L_s) and (L_t)), computes their intersection, and forms a new clique ((L_s \cap L_t) \cup {s,t}). Every subset of this new clique that includes the edge corresponds to a new simplex that has not existed before. Proposition 2.1 rigorously proves that (i) this construction yields all new simplices, (ii) no existing maximal cliques are lost, and (iii) the set of new simplices is exactly the set of cliques that contain the new edge. Consequently, the simplex stream can be generated on‑the‑fly, one edge at a time, without ever materialising the full complex.

The persistence algorithm (originally by Edelsbrunner‑Zomorodian and refined by Zomorodian‑Carlsson) consumes a simplex stream while maintaining an internal state that records both completed intervals and those still open. By feeding the on‑the‑fly simplices directly into this algorithm, the computation proceeds interleaved: each new edge triggers immediate updates to the homology, and the current state can be inspected at any moment. This eliminates the need for a separate “pre‑processing” phase and enables interactive exploration.

Memory analysis shows two dominant contributors. First, the temporary storage needed for the incremental Vietoris‑Rips construction is proportional to the number of new cofaces generated per edge, which the authors bound by (O(nk)) where (n) is the number of vertices and (k) the maximal simplex dimension of interest. Second, the persistence table that stores marked simplices and their cascades also requires (O(nk)) space in the worst case, but empirical evidence suggests the cascade depth is modest, reducing the constant factor. Overall, the algorithm’s memory footprint scales linearly with the number of vertices for a fixed (k), a substantial improvement over the exponential growth of naïve constructions.

To further stretch the limits of available hardware, the paper advocates a hierarchical storage strategy. The full distance matrix ((O(n^2)) entries) can be streamed to disk as it is computed, sorted in‑place, and later read incrementally to produce the edge stream. Likewise, once a persistence interval is closed, its full interval data can be off‑loaded to secondary storage, retaining only the minimal information required for future updates. By relegating the distance graph and completed intervals to disk, RAM usage can be kept within the range of a few gigabytes even for datasets that would otherwise demand terabytes of memory.

The authors illustrate the method with a small example graph, showing how new maximal cliques arise from edge insertions, how redundant simplices are eliminated, and how subsets of the intersected cliques generate the full set of new faces. Pseudocode outlines the processing steps: update maximal cliques, compute intersections, generate new simplices, deduplicate, and feed them to the persistence engine.

In summary, the paper contributes three key innovations: (1) an incremental, maximal‑clique‑driven construction of Vietoris‑Rips simplices; (2) a tightly coupled, interleaved execution with a standard persistence algorithm; and (3) a pragmatic memory‑management scheme that leverages disk storage to handle large distance graphs and completed intervals. These ideas together enable persistent homology computations on data sets that exceed the memory capacities of existing software, while preserving the ability to inspect intermediate results and to continue computation without restarting. Future work is suggested in optimizing maximal‑clique maintenance (e.g., using Bron‑Kerbosch variants or parallel updates), quantifying cascade size distributions, and benchmarking the approach on real‑world high‑dimensional data.