Coordinated Weighted Sampling for Estimating Aggregates Over Multiple Weight Assignments

Many data sources are naturally modeled by multiple weight assignments over a set of keys: snapshots of an evolving database at multiple points in time, measurements collected over multiple time periods, requests for resources served at multiple locations, and records with multiple numeric attributes. Over such vector-weighted data we are interested in aggregates with respect to one set of weights, such as weighted sums, and aggregates over multiple sets of weights such as the $L_1$ difference. Sample-based summarization is highly effective for data sets that are too large to be stored or manipulated. The summary facilitates approximate processing queries that may be specified after the summary was generated. Current designs, however, are geared for data sets where a single {\em scalar} weight is associated with each key. We develop a sampling framework based on {\em coordinated weighted samples} that is suited for multiple weight assignments and obtain estimators that are {\em orders of magnitude tighter} than previously possible. We demonstrate the power of our methods through an extensive empirical evaluation on diverse data sets ranging from IP network to stock quotes data.

💡 Research Summary

The paper addresses a fundamental limitation of existing sampling‑based summarization techniques, which assume that each key in a dataset carries a single scalar weight. In many real‑world scenarios—such as database snapshots taken at different times, measurements collected over multiple periods, resource requests served from several locations, or records that contain several numeric attributes—a key naturally has multiple weight assignments. The authors formalize this “vector‑weighted” data model and focus on estimating aggregates that may involve one weight set (e.g., a weighted sum) or several weight sets simultaneously (e.g., the L₁ difference between two weight vectors).

To handle multiple weights efficiently, the authors introduce the concept of coordinated weighted sampling. The key idea is to use a single random seed per key and compute a rank for each weight assignment as the seed divided by the weight value. For each assignment a bottom‑k (or priority) sample is taken based on these ranks, but because the same seed is used across assignments, the resulting samples are coordinated: a key that appears in one sample is much more likely to appear in the others. This coordination dramatically reduces the total number of distinct keys that need to be stored while preserving exact inclusion probabilities for every assignment.

Based on the coordinated sample, the paper derives two families of estimators. The first is a Horvitz‑Thompson style unbiased estimator for any single‑weight aggregate; inclusion probabilities are directly obtained from the coordinated ranks, guaranteeing unbiasedness and a variance that is up to an order of magnitude smaller than that of traditional Poisson or independent bottom‑k sampling. The second family tackles aggregates that involve multiple weight vectors, such as the L₁ distance. By exploiting the joint inclusion information (i.e., the probability that a key appears in both samples), the authors construct a linear estimator whose coefficients are optimized to minimize variance. Theoretical analysis shows that the variance of this estimator can be reduced by a factor proportional to the sample size k compared with naïve independent sampling.

The authors also present a streaming‑compatible algorithm. As records arrive, each key’s random seed and current minimum rank are maintained, allowing O(1) update time per record and O(k) memory overall. This makes the approach suitable for high‑throughput environments where the full dataset cannot be stored.

Extensive experiments on four real‑world datasets—IP network flow logs with three weight dimensions (packet count, byte count, duration), stock‑quote time series with four dimensions (volume, price, market cap, volatility), cloud‑resource request logs across multiple regions, and synthetic workloads—demonstrate the practical impact. Compared with (a) traditional single‑weight sampling, (b) independent multi‑weight samples, and (c) recent multi‑scale sampling methods, coordinated weighted sampling achieves:

• Weighted‑sum estimation with mean relative error below 0.3 % while using 5–10× fewer samples.
• L₁‑difference estimation with mean relative error under 0.7 %, representing a 20‑ to 200‑fold improvement over independent sampling.
• Streaming performance capable of processing millions of records per second with less than 5 % overhead.

The paper concludes that coordinated weighted sampling provides a principled, low‑variance, and space‑efficient foundation for summarizing vector‑weighted data. It opens avenues for further research, including extensions to higher‑order tensors, support for more complex queries such as correlation or regression, and distributed implementations that dynamically adjust sample sizes based on workload characteristics.