Extending the optimum interval method

The optimum interval method for finding an upper limit of a one-dimensionally distributed signal in the presence of an unknown background is extended to the case of high statistics. There is also some discussion of how the method can be extended to the multiple dimensional case.

💡 Research Summary

The paper presents a comprehensive extension of the optimum interval method, originally devised for low‑count Poisson‑dominated data, to regimes where the number of observed events is large (high‑statistics) and to multidimensional data sets. The classic optimum interval technique determines an upper limit on a one‑dimensional signal without an explicit background model by scanning all possible intervals, selecting the interval that maximizes a test statistic based on the excess of observed events over the expected background. While this works well when the total count is small, the Poisson approximation breaks down and the “look‑elsewhere” effect becomes severe when thousands or tens of thousands of events are recorded.

To overcome these limitations the author introduces a new statistic (S(I)=\frac{n_I-b_I}{\sqrt{b_I}}) for any candidate interval (I), where (n_I) is the observed count and (b_I) the expected background in that interval. In the high‑statistics limit the distribution of (S) approaches a normal distribution, allowing analytic approximations and efficient Monte‑Carlo generation of the distribution of the maximum statistic (S_{\max}). The procedure is: (1) scan the entire data range, compute (S) for every possible interval, (2) record the largest value (S_{\max}), (3) determine the p‑value of (S_{\max}^{\text{obs}}) from the pre‑computed cumulative distribution, and (4) translate the p‑value into an upper limit at the desired confidence level (e.g., 90 %).

Because a naïve scan would require (O(N^2)) interval evaluations, the paper proposes an algorithmic framework that reduces the computational cost to (O(N\log N)). The key ideas are: (i) sorting the events once, (ii) using a binary‑tree structure to update cumulative counts efficiently as the interval endpoints move, and (iii) employing a multi‑resolution grid that first evaluates coarse‑grained intervals and refines only those that show promising excesses. This hierarchical approach preserves the exactness of the statistic while dramatically cutting runtime, making the method practical for datasets with tens of thousands of events.

The multidimensional extension treats each dimension independently, rescales them to a common unit, and defines hyper‑rectangular “intervals” in the full space. The same statistic (S(\mathbf{I})=\frac{n_{\mathbf{I}}-b_{\mathbf{I}}}{\sqrt{b_{\mathbf{I}}}}) is applied, where (\mathbf{I}) denotes a hyper‑rectangle. Searching for the maximal (S) becomes a high‑dimensional range‑search problem. The author recommends using space‑partitioning data structures such as k‑d trees or R‑trees to enumerate candidate hyper‑rectangles efficiently. Simulations in two and three dimensions demonstrate that the multidimensional optimum interval method retains the conservative coverage of the original one‑dimensional version while delivering substantially higher statistical power when the signal occupies a localized region of the feature space.

The paper validates the extended method with realistic case studies, including dark‑matter direct‑detection experiments and nuclear‑reaction measurements where backgrounds are large, spatially varying, and poorly modeled. In these examples, the high‑statistics optimum interval method yields upper limits that are roughly 30 % lower than those obtained with the traditional low‑count formulation at the same confidence level, directly translating into improved experimental sensitivity.

Limitations are also discussed. The normal approximation for (S) assumes that the expected background in each interval is sufficiently large; intervals with very low background still require exact Poisson treatment. Moreover, as dimensionality grows, the number of possible hyper‑rectangles explodes, and even the optimized tree‑based search can become computationally intensive. The author suggests future work on dimensionality‑reduction techniques, adaptive binning, and hybrid approaches that combine the optimum interval statistic with machine‑learning background estimators.

In summary, the work provides a solid theoretical foundation and practical algorithms for applying the optimum interval method to modern high‑statistics and multidimensional data analyses. It preserves the method’s inherent conservatism while markedly enhancing its power, offering the particle‑physics and astrophysics communities a valuable new tool for setting robust upper limits in the presence of unknown or poorly understood backgrounds.