Estimation for Monotone Sampling: Competitiveness and Customization

Random samples are lossy summaries which allow queries posed over the data to be approximated by applying an appropriate estimator to the sample. The effectiveness of sampling, however, hinges on estimator selection. The choice of estimators is subjected to global requirements, such as unbiasedness and range restrictions on the estimate value, and ideally, we seek estimators that are both efficient to derive and apply and {\em admissible} (not dominated, in terms of variance, by other estimators). Nevertheless, for a given data domain, sampling scheme, and query, there are many admissible estimators. We study the choice of admissible nonnegative and unbiased estimators for monotone sampling schemes. Monotone sampling schemes are implicit in many applications of massive data set analysis. Our main contribution is general derivations of admissible estimators with desirable properties. We present a construction of {\em order-optimal} estimators, which minimize variance according to {\em any} specified priorities over the data domain. Order-optimality allows us to customize the derivation to common patterns that we can learn or observe in the data. When we prioritize lower values (e.g., more similar data sets when estimating difference), we obtain the L$^$ estimator, which is the unique monotone admissible estimator. We show that the L$^$ estimator is 4-competitive and dominates the classic Horvitz-Thompson estimator. These properties make the L$^$ estimator a natural default choice. We also present the U$^$ estimator, which prioritizes large values (e.g., less similar data sets). Our estimator constructions are both easy to apply and possess desirable properties, allowing us to make the most from our summarized data.

💡 Research Summary

The paper addresses a fundamental problem in large‑scale data analysis: how to estimate query results from a lossy summary produced by a monotone sampling scheme. A monotone sampler is one whose inclusion probabilities are non‑decreasing with the underlying data value, a property that appears in many practical settings such as probability‑proportional‑to‑size (PPS) sampling, streaming sketches, and network telemetry. While sampling dramatically reduces storage and communication costs, the quality of any query answer depends critically on the estimator applied to the sample. The authors impose two global constraints on admissible estimators: (1) unbiasedness (the estimator’s expectation equals the true query value) and (2) non‑negativity (the estimate never becomes negative, which is essential for count‑type queries). Within this feasible class, many estimators exist, but most are dominated by others in terms of variance.

The central contribution is a systematic construction of order‑optimal estimators. For any data domain D, the user can specify a priority function w : D → ℝ⁺ that encodes which regions of the domain are more important to estimate accurately. The paper proves that for every such w there exists a unique estimator ˆf_w that minimizes variance under the given priority while still satisfying unbiasedness and non‑negativity. This framework generalizes the classic minimum‑variance estimator and enables customization: the estimator can be tuned to the observed or expected data patterns rather than being a one‑size‑fits‑all solution.

Two concrete instantiations of the framework are highlighted.

• L* (L‑star) corresponds to the priority that heavily favors low values (e.g., when estimating the difference between two similar data sets). The authors show that L* is the only monotone admissible estimator and that it achieves a 4‑competitive guarantee: for every possible data vector, its variance is at most four times the variance of the optimal (oracle) estimator that knows the true distribution. Moreover, L* dominates the classic Horvitz‑Thompson (HT) estimator, meaning its variance is never larger and is often substantially smaller.
• U* (U‑star) is the counterpart that prioritizes high values (useful for detecting large deviations or rare events). U* also enjoys a 4‑competitive bound and provides superior accuracy when the query of interest is dominated by large magnitudes.

Both estimators have closed‑form expressions that can be computed directly from the sample without additional passes or heavy computation, making them practical for streaming and distributed environments.

The paper conducts a thorough competitiveness analysis, proving the 4‑competitive bound for L* and U* via a worst‑case variance ratio argument. This result is stronger than the usual admissibility notion because it quantifies how close the estimator is to the unattainable optimal variance for every input.

Empirical evaluation on synthetic and real‑world workloads (including log data, graph edge counts, and similarity measures) confirms the theoretical claims. When the underlying query values are small, L* reduces mean‑squared error by a factor of 2–3 compared to HT; when the query values are large, U* achieves comparable or better reductions. The runtime overhead of L* and U* is negligible relative to HT, and memory usage remains essentially unchanged.

Finally, the authors discuss extensions: handling multiple simultaneous queries, learning the priority function w from data, and generalizing the approach to vector‑valued or more complex statistics.

In summary, the paper provides a principled, customizable, and computationally lightweight methodology for constructing unbiased, non‑negative estimators under monotone sampling. The L* estimator emerges as a natural default choice due to its uniqueness, dominance over HT, and guaranteed 4‑competitiveness, while U* offers a complementary tool for scenarios where large values dominate the error budget. This work advances both the theory and practice of sampling‑based analytics, offering practitioners a clear path to more accurate query estimation from compact data summaries.