Optimally detecting uniformly-distributed $ll_2$ heavy hitters in data streams

Given a stream $x_1,x_2,\dots,x_n$ of items from a Universe $U$ of size poly$(n)$, and a parameter $ε>0$, an item $i\in U$ is said to be an $\ell_2$ heavy hitter if its frequency $f_i$ in the stream is at least $\sqrt{εF_2}$, where $F_2={\sum_{i\in U} f_i^2}$. Efficiently detecting such heavy hitters is a fundamental problem in data streams and has several applications in both theory and in practice. The classical $\mathsf{CountSketch}$ algorithm due to Charikar, Chen, and Farach-Colton [2004], was the first algorithm to detect $\ell_2$ heavy hitters using $O\left(\frac{\log^2 n}ε\right)$ bits of space, and their algorithm is optimal for streams with deletions. A work due to Braverman, Chestnut, Ivkin, Nelson, Wang, and Woodruff [2017] gave the $\mathsf{BPTree}$ algorithm which detects $\ell_2$ heavy hitters in insertion-only streams using only $O\left(\frac{\log(1/ε)}ε\log n \right)$ space. Note that any algorithm requires at least $Ω\left(\frac{1}ε \log n\right)$ space to output $O(1/ε)$ heavy hitters in the worst case. While $\mathsf{BPTree}$ achieves optimal space bound for constant $ε$, their bound could be sub-optimal for $ε=o(1)$. For $\textit{random order}$ streams, where the stream elements can be adversarial but their order of arrival is uniformly random, Braverman, Garg, and Woodruff [2020] showed that it is possible to achieve the optimal space bound of $O\left(\frac{1}ε \log n\right)$ for every $ε= Ω\left(\frac{1}{2^{\sqrt{\log n}}}\right)$. In this work, we generalize their result to $\textit{partially random order}$ streams where only the heavy hitters are required to be uniformly distributed in the stream. We show that it is possible to achieve the same space bound, but with an additional assumption that the algorithm is given a constant approximation to $F_2$ in advance.

💡 Research Summary

The paper studies the problem of identifying ℓ₂‑heavy hitters in data streams, where an item i is an ℓ₂‑heavy hitter if its frequency f_i satisfies f_i ≥ √(ε F₂) with F₂ = Σ_j f_j². Classical solutions such as CountSketch achieve O((log² n)/ε) space and are optimal for turnstile (insertion‑deletion) streams. For insertion‑only streams, CountSieve and BPTree improve the space to O((log(1/ε))/ε·log n·log log n) and O((log(1/ε))/ε·log n) respectively, but still contain a log(1/ε) factor that becomes significant when ε is very small. In the random‑order model, Braverman, Garg, and Woodruff showed that O((1/ε)·log n) space is sufficient for ε ≥ Ω(1/2^{√log n}), matching the lower bound Ω((1/ε)·log n).

The contribution of this work is to extend the optimal O((1/ε)·log n) bound to a “partially random order” setting. In this model the adversary may choose the order of all non‑heavy items, but the heavy hitters themselves appear uniformly at random throughout the stream. Additionally, the algorithm is given a constant‑factor approximation to F₂ in advance and has access to a random oracle (i.e., truly random hash functions). Under these assumptions, the authors present a single‑pass streaming algorithm that, with probability at least 0.9, reports every ℓ₂‑ε‑heavy hitter and no item that is not an ℓ₂‑(ε/256)‑heavy hitter, using only O((1/ε)·log n) bits of space for any ε ≥ c/√log n (for some absolute constant c).

Technical Overview
The algorithm’s core is a hierarchical sample‑and‑check framework built on a carefully chosen window size. The stream of length n is partitioned into windows of length
 W ≈ n·√(ε/F₂).
Because heavy hitters are uniformly distributed, with high probability each heavy hitter appears in every block of O(log n) consecutive windows. For each window i the algorithm samples a subset S_i of the universe, where each element is included independently with probability q = 1/W. The expected number of sampled items per window is constant. When a window contains exactly one sampled item x_i, the algorithm stores only the hash value v = h(x_i) (h maps to a polylog n range). In the next window(s) the algorithm checks whether any sampled element shares the same hash v and also belongs to S_i. If such a match occurs, the algorithm “promotes” the candidate, reads the full identifier of the element (costing O(log n) bits), and continues checking across a few more windows to confirm that the candidate appears repeatedly—characteristic of a true heavy hitter.

To keep space low, the algorithm runs many parallel instances of this procedure (≈ O(log n / log log n) instances) but enforces that at most one instance may allocate O(log n) bits at any time. The other instances operate in a “paused” mode, consuming only O(log log n) bits (the hash value). Because the probability that a non‑heavy item triggers a full‑identifier read is only 1/K (with K = polylog n), the fraction of time any instance needs the larger memory is negligible. Consequently, the total space stays within O((1/ε)·log n).

Correctness arguments rely on two observations. First, any true heavy hitter is guaranteed to be the unique sampled item in a constant‑fraction of windows, and its hash will reappear in the next window with probability 1, leading to successful promotion. Second, for any light item, the chance of repeatedly matching the same hash across consecutive windows is bounded by 1/K per window, so with high probability the candidate is discarded after O(1) windows. By union‑bounding over all O(1/ε) heavy hitters and using the parallel instances, the algorithm achieves the desired success probability.

The requirement of a constant‑factor estimate of F₂ is technical: the window length W depends on √(ε/F₂). Without this estimate, one would need to guess F₂ geometrically, incurring an extra O(log (1/ε)) factor, which would destroy the optimality. The authors note that integrating an ℓ₂‑norm tracker (as in BPTree) into the partially random order model seems to require additional space, leaving the removal of this assumption as an open problem.

Significance
This work bridges the gap between fully random‑order streams and adversarial streams by showing that a modest randomness assumption—uniform distribution of heavy hitters—suffices to achieve the optimal space bound for ℓ₂‑heavy hitter detection. It eliminates the log(1/ε) overhead present in the best known insertion‑only algorithms, matching the lower bound for a wide range of ε (down to Θ(1/√log n)). The hierarchical sample‑and‑check technique and the dynamic time‑sharing of memory across parallel instances may inspire further streaming algorithms under hybrid randomness models. Future directions include removing the F₂ approximation requirement and extending the approach to other norms or to settings with multiple heavy‑hitters per window.