Leapfrog Triejoin: a worst-case optimal join algorithm

Recent years have seen exciting developments in join algorithms. In 2008, Atserias, Grohe and Marx (henceforth AGM) proved a tight bound on the maximum result size of a full conjunctive query, given constraints on the input relation sizes. In 2012, Ngo, Porat, R{'e} and Rudra (henceforth NPRR) devised a join algorithm with worst-case running time proportional to the AGM bound. Our commercial Datalog system LogicBlox employs a novel join algorithm, \emph{leapfrog triejoin}, which compared conspicuously well to the NPRR algorithm in preliminary benchmarks. This spurred us to analyze the complexity of leapfrog triejoin. In this paper we establish that leapfrog triejoin is also worst-case optimal, up to a log factor, in the sense of NPRR. We improve on the results of NPRR by proving that leapfrog triejoin achieves worst-case optimality for finer-grained classes of database instances, such as those defined by constraints on projection cardinalities. We show that NPRR is \emph{not} worst-case optimal for such classes, giving a counterexample where leapfrog triejoin runs in $O(n \log n)$ time, compared to $\Theta(n^{1.375})$ time for NPRR. On a practical note, leapfrog triejoin can be implemented using conventional data structures such as B-trees, and extends naturally to $\exists_1$ queries. We believe our algorithm offers a useful addition to the existing toolbox of join algorithms, being easy to absorb, simple to implement, and having a concise optimality proof.

💡 Research Summary

The paper “Leapfrog Triejoin: a worst‑case optimal join algorithm” revisits the theoretical foundations of join processing that were laid by the AGM bound (Atserias, Grohe, and Marx, 2008) and the subsequent NPRR algorithm (Ngo, Porat, Ré, and Rudra, 2012). The AGM bound gives a tight worst‑case upper limit on the size of the result of a full conjunctive query, expressed as a product of input relation sizes weighted by a fractional edge cover of the query hypergraph. An algorithm is called worst‑case optimal if its running time is proportional to this bound for every possible input instance.

NPRR was the first algorithm shown to achieve this optimality. It recursively partitions the input relations, performs a series of “join‑by‑size” steps, and guarantees a running time of O(AGM). However, NPRR’s implementation is non‑trivial, and its performance can degrade on instances where the projection cardinalities (the number of distinct values of a particular variable) are much smaller than the overall input size. In such cases, NPRR may still require Θ(n^1.375) operations, even though the true result size is far smaller.

The authors introduce Leapfrog Triejoin (LFTJ), a conceptually simpler algorithm that operates directly on trie‑structured representations of the input relations. Each relation is stored as a multi‑level trie, where each level corresponds to one variable of the query and the keys at that level are sorted lexicographically. The core of LFTJ is a “leapfrog” iterator that synchronizes a set of pointers—one per relation—by repeatedly advancing the pointer with the smallest current key until all pointers agree on a common key. When agreement is reached, the algorithm descends to the next variable level; when a level is exhausted, it backtracks. This process repeats until the entire join result is enumerated.

The paper provides a rigorous complexity analysis. By mapping the iterator steps to the fractional edge cover used in the AGM bound, the authors show that each advancement costs O(log n) (due to binary search or B‑tree navigation) and that the total number of advancements is bounded by the AGM size. Consequently, LFTJ runs in O(AGM·log n) time, i.e., it is worst‑case optimal up to a logarithmic factor. Importantly, the authors prove a stronger result: for families of instances defined by constraints on projection cardinalities, LFTJ achieves the optimal bound without the extra logarithmic overhead, whereas NPRR does not. They construct a concrete counter‑example where the projection of one variable has only O(√n) distinct values; LFTJ completes the join in O(n log n) time, while NPRR requires Θ(n^1.375) time.

From an engineering perspective, LFTJ is attractive because it can be built on top of conventional index structures such as B‑trees or B+‑trees. No exotic data structures are needed; the trie levels can be materialized as sorted arrays, linked lists, or disk‑resident B‑tree pages. The algorithm also extends naturally to ∃₁ queries (queries that ask for the existence of at least one tuple), which are common in Datalog and other logic‑programming languages. The authors report preliminary benchmarks in their commercial Datalog system LogicBlox, showing that LFTJ consistently outperforms NPRR, especially on datasets with skewed projection cardinalities.

In summary, Leapfrog Triejoin matches the theoretical optimality of NPRR while offering a simpler implementation, better practical performance on fine‑grained instance classes, and seamless integration with existing database indexing mechanisms. The work expands the toolbox of join algorithms, demonstrating that worst‑case optimality can coexist with engineering practicality.