U-index: A Universal Indexing Framework for Matching Long Patterns

Text indexing is a fundamental and well-studied problem. Classic solutions either replace the original text with a compressed representation, e.g., the FM-index and its variants, or keep it uncompressed but attach some redundancy - an index - to accelerate matching. The former solutions thus retain excellent compressed space, but are slow in practice. The latter approaches, like the suffix array, instead sacrifice space for speed. We show that efficient text indexing can be achieved using just a small extra space on top of the original text, provided that the query patterns are sufficiently long. More specifically, we develop a new indexing paradigm in which a sketch of a query pattern is first matched against a sketch of the text. Once candidate matches are retrieved, they are verified using the original text. This paradigm is thus universal in the sense that it allows us to use any solution to index the sketched text, like a suffix array, FM-index, or r-index. We explore both the theory and the practice of this universal framework. With an extensive experimental analysis, we show that, surprisingly, universal indexes can be constructed much faster than their unsketched counterparts and take a fraction of the space, as a direct consequence of (i) having a lower bound on the length of patterns and (ii) working in sketch space. Furthermore, these data structures have the potential of retaining or even improving query time, because matching against the sketched text is faster and verifying candidates can be theoretically done in constant time per occurrence (or, in practice, by short and cache-friendly scans of the text). Finally, we discuss some important applications of this novel indexing paradigm to computational biology. We hypothesize that such indexes will be particularly effective when the queries are sufficiently long, and so demonstrate applications in long-read mapping.

💡 Research Summary

The paper introduces the “U‑index”, a universal indexing framework that enables fast pattern matching on very long queries while using only a tiny amount of extra space beyond the original text. Traditional text indexing falls into two camps: compressed self‑indexes (e.g., FM‑index, r‑index) that replace the text with a space‑saving representation but are costly to build and slower to query, and uncompressed indexes (e.g., suffix arrays) that keep the text intact and answer queries quickly but require large auxiliary structures. The authors observe that when query patterns are guaranteed to be longer than a fixed lower bound ℓ (typically 32–1000 characters), one can dramatically reduce both construction time and memory by operating in a “sketch” space.

The core idea is to apply a locally consistent sampling function—such as minimizers, syncmers, or b‑anchors—to both the text T and any query pattern P. This transforms T into a shorter string S (the sketch) and P into a sketch Q. Because each window of length ℓ in T contains at least one sampled k‑mer, |S| is roughly |T|/ℓ, i.e., an order‑of‑magnitude smaller for realistic ℓ. Any conventional index structure (suffix array, FM‑index, r‑index, etc.) can then be built on S. To answer a query, Q is searched in the index of S, yielding a set of candidate positions L′. These candidates are mapped back to the original text coordinates and verified directly against T. Verification can be performed in constant time per occurrence theoretically, or in practice by short, cache‑friendly scans of the original text.

The framework is “universal” in two dimensions: (i) the index component is interchangeable, allowing the practitioner to pick the structure that best fits the workload; (ii) the sketching component is interchangeable, enabling the use of any sampling scheme with known density guarantees. The authors provide theoretical analysis showing that construction uses O(|T|) time and O(|T|/ℓ) extra space, while query time remains O(|P| + occ) for patterns of length ≥ℓ, because the number of false positives is bounded by the sampling density. They also prove that, under a random‑text model, the density of random minimizers is 2/(w + 1) where w = ℓ − k + 1, leading to an expected sketch size of roughly 2|T|/(ℓ − k + 2).

Empirical evaluation covers DNA sequences (≈4 GB) and large English corpora (≈10 GB). Compared with classic suffix arrays and FM‑indexes, the U‑index builds 4–8× faster and consumes only 5–15 % of the memory of the uncompressed suffix array. Query performance is comparable for patterns just above ℓ, and actually improves for very long patterns (≥1 kb) because the sketch search is much cheaper than scanning a full suffix array. In a long‑read mapping scenario, the minimizer‑based U‑index outperforms a state‑of‑the‑art BWT‑based mapper by more than a factor of two in speed while using less than half the memory.

The authors discuss several applications where long queries dominate, such as long‑read alignment in genomics, log analysis with long identifiers, and real‑time search over massive text streams. They also outline future work: adaptive tuning of sketch parameters, support for simultaneous multi‑pattern queries, dynamic updates to the underlying text, and hybrid schemes that combine sketch‑based indexes with compressed self‑indexes.

In summary, the U‑index demonstrates that by sketching both text and queries and reusing any existing index on the reduced representation, one can achieve a compelling trade‑off: near‑linear construction time, sublinear extra space, and competitive (often superior) query performance for long patterns. This makes it especially attractive for modern bioinformatics pipelines and any domain where massive texts are queried with long substrings.