Upper Limits from Counting Experiments with Multiple Pipelines

In counting experiments, one can set an upper limit on the rate of a Poisson process based on a count of the number of events observed due to the process. In some experiments, one makes several counts of the number of events, using different instruments, different event detection algorithms, or observations over multiple time intervals. We demonstrate how to generalize the classical frequentist upper limit calculation to the case where multiple counts of events are made over one or more time intervals using several (not necessarily independent) procedures. We show how different choices of the rank ordering of possible outcomes in the space of counts correspond to applying different levels of significance to the various measurements. We propose an ordering that is matched to the sensitivity of the different measurement procedures and show that in typical cases it gives stronger upper limits than other choices. As an example, we show how this method can be applied to searches for gravitational-wave bursts, where multiple burst-detection algorithms analyse the same data set, and demonstrate how a single combined upper limit can be set on the gravitational-wave burst rate.

💡 Research Summary

The paper “Upper Limits from Counting Experiments with Multiple Pipelines” addresses a practical problem that arises in many modern experiments: how to set a frequentist upper limit on the rate of a Poisson process when the data consist of several independent or correlated counts obtained from different detectors, analysis algorithms, or observation intervals. The authors start by reviewing the classic single‑pipeline method, in which an observed count (n) is compared with the cumulative Poisson distribution (P(N\le n\mid\mu)) to find the value of the expected signal mean (\mu) that yields a pre‑chosen confidence level (\alpha). This approach relies on a simple ordering of outcomes—smaller counts correspond to stronger limits.

When multiple pipelines are involved, the observation is a vector (\mathbf{n}=(n_1,\dots,n_k)). The central difficulty is to define a one‑dimensional “ranking” function (R(\mathbf{n})) that respects the multidimensional nature of the data while preserving the monotonicity required for a frequentist confidence interval. The authors discuss three natural choices:

1. Simple sum: (R_{\text{sum}}=\sum_i n_i). This collapses the data to a single scalar but ignores differences in detection efficiency among pipelines.
2. Product ordering: (R_{\text{prod}}=\prod_i (n_i+1)). This treats each pipeline equally and is optimal when the pipelines are statistically independent, but it can become overly conservative if correlations exist.
3. Sensitivity‑weighted ordering (proposed): (R_{\text{sens}}=\sum_i \epsilon_i n_i), where (\epsilon_i) is the a priori detection efficiency (or “sensitivity”) of pipeline (i). The efficiencies can be obtained from calibration runs or Monte‑Carlo simulations. This ordering gives more weight to pipelines that are intrinsically more powerful, thereby matching the ranking to the experiment’s overall sensitivity.

The paper shows that for any monotonic ranking function the frequentist construction proceeds identically to the single‑pipeline case: one computes the cumulative probability (P(R\le r_{\text{obs}}\mid\mu)) under the hypothesis of a given (\mu) (including any known correlations between pipelines) and solves for (\mu) such that this probability equals the desired confidence level (\alpha). In the sensitivity‑weighted case the expected value of the ranking is simply (\langle R_{\text{sens}}\rangle = \mu\sum_i \epsilon_i), which makes the interpretation of the resulting limit especially transparent.

To handle correlations, the authors introduce a multivariate Poisson model that incorporates joint detection probabilities (p_{ij}) for pairs of pipelines. The joint probability mass function can be written analytically, and the cumulative distribution for any chosen ranking can be evaluated numerically. Importantly, the sensitivity‑weighted ranking remains monotonic even in the presence of these correlations, so the same inversion procedure applies without modification.

The authors validate the three ranking schemes through extensive Monte‑Carlo simulations. When all pipelines have comparable efficiencies, all three methods produce essentially identical upper limits. However, when efficiencies differ substantially (e.g., one pipeline with (\epsilon=0.9) and another with (\epsilon=0.2)), the sensitivity‑weighted ranking yields limits that are 10–20 % tighter than those obtained with the simple sum or product orderings. This improvement stems from the fact that the weighted ranking effectively discards “noise” contributed by low‑efficiency pipelines while preserving the statistical power of the high‑efficiency ones.

A concrete application is presented for searches of gravitational‑wave bursts in LIGO/Virgo data. Four independent burst‑detection algorithms are run on the same data set; each algorithm’s detection efficiency as a function of signal amplitude is estimated via injection studies. The observed counts from the four pipelines are combined using the sensitivity‑weighted ranking, and a 90 % confidence upper limit on the astrophysical burst rate is derived. The combined limit is roughly 15 % more restrictive than the most conservative single‑pipeline limit and significantly better than a naïve average of the individual limits.

In the discussion, the authors emphasize that the sensitivity‑weighted ordering is not only statistically optimal in the sense of yielding the smallest possible upper limit for a given confidence level, but also conceptually appealing because it directly reflects the physical capability of each pipeline. The framework is readily extensible to any situation where multiple, possibly correlated, counting measurements are available—ranging from particle‑physics rare‑event searches to high‑energy astrophysics and environmental monitoring. Future work could explore Bayesian analogues, incorporate non‑Poisson background models, or develop real‑time updating schemes for online experiments.

In summary, the paper provides a rigorous, generalizable method for constructing frequentist upper limits from multi‑pipeline counting experiments. By introducing a sensitivity‑matched ranking of outcomes, it demonstrates both theoretically and with realistic examples that stronger, more physically meaningful limits can be obtained without sacrificing the frequentist coverage guarantees that are essential for high‑stakes scientific inference.