Statistical methods used in ATLAS for exclusion and discovery

The statistical methods used by the ATLAS Collaboration for setting upper limits or establishing a discovery are reviewed, as they are fundamental ingredients in the search for new phenomena. The analyses published so far adopted different approaches, choosing a frequentist or a Bayesian or a hybrid frequentist-Bayesian method to perform a search for new physics and set upper limits. In this note, after the introduction of the necessary basic concepts of statistical hypothesis testing, a few recommendations are made about the preferred approaches to be followed in future analyses.

💡 Research Summary

The note provides a comprehensive review of the statistical techniques employed by the ATLAS Collaboration for setting exclusion limits and claiming discoveries, and it puts forward concrete recommendations for future analyses. After a brief overview of the ATLAS simulation chain—event generation, detector response simulation, and digitisation—the authors discuss the various sources of statistical and systematic uncertainties that arise at each stage, emphasizing that differences between event generators should be treated as independent systematic effects rather than combined in quadrature.

Section 2 introduces the basic concepts of hypothesis testing in high‑energy physics. In the frequentist framework, a p‑value is defined as the probability, under the null hypothesis, of obtaining data at least as extreme as observed; this is often translated into a Z‑score using the inverse Gaussian cumulative distribution. A discovery requires Z ≥ 5 (p ≤ 2.87 × 10⁻⁷), while exclusion limits are typically quoted at the 95 % confidence level (α = 0.05). The Neyman–Pearson approach is also described, with explicit definitions of test size (α) and power (1 – β). The Bayesian perspective is presented in parallel, focusing on posterior odds, Bayes factors, and the role of prior information.

Section 3 surveys the methods actually used in past ATLAS publications. Early analyses relied heavily on the CLs technique, which rejects the signal‑plus‑background hypothesis if the ratio CLs = p_{s+b}/(1 – p_b) falls below a chosen α. Later, Power‑Constrained Limits (PCL) were introduced to enforce a minimum power requirement alongside the p‑value cut. Systematic uncertainties have increasingly been handled with the HistFactory tool built on RooFit/RooStats, allowing simultaneous profiling of many nuisance parameters. Specialized tools such as BumpHunter have been employed for “bump‑search” scenarios where the signal location and width are unknown. A hybrid Bayesian‑frequentist approach has also appeared in some Higgs combination studies, where nuisance parameters receive Bayesian priors while the parameter of interest is treated with frequentist p‑values.

Section 4 contains the core recommendations. The authors advocate the use of a single, unified frequentist method for all ATLAS searches. The model assumes that the expected event count in bin i is E(n_i) = μ s_i + b_i, where μ is the signal strength multiplier, s_i the expected signal yield, and b_i the background expectation. For discovery, the null hypothesis μ = 0 (background‑only) is tested; if no significant excess is found, an upper limit on μ is set. The profile likelihood ratio is the test statistic, and in the asymptotic regime its distribution can be approximated analytically. The Asimov dataset—an idealised Monte‑Carlo sample that yields the true parameter values when fitted—is used to compute the maximum‑likelihood estimate \hat{μ} and its standard deviation σ. Approximate exclusion limits are then \hat{μ} ± σ Φ⁻¹(1 – α/2). For discovery sensitivity, the median significance is given by
 Z_med = √{2