Testing the approximations described in "Asymptotic formulae for likelihood-based tests of new physics"

“Asymptotic formulae for likelihood-based tests of new physics” presents a mathematical formalism for a new approximation for hypothesis testing in high energy physics. The approximations are designed to greatly reduce the computational burden for such problems. We seek to test the conditions under which the approximations described remain valid. To do so, we perform parallel calculations for a range of scenarios and compare the full calculation to the approximations to determine the limits and robustness of the approximation. We compare this approximation against values calculated with the Collie framework, which for our analysis we assume produces true values.

💡 Research Summary

The paper investigates the validity of the asymptotic approximations introduced in the seminal work “Asymptotic formulae for likelihood‑based tests of new physics.” These approximations—collectively referred to as the A‑WW approximation (Asimov data set, Wald equation, and Tevatron test statistic)—are designed to replace computationally intensive Monte‑Carlo based hypothesis testing with analytic formulas that are fast to evaluate. The authors set out to delineate the regime in which these approximations remain accurate by comparing them against a full semi‑frequentist calculation performed with the Collie framework, which they treat as the ground‑truth.

The manuscript begins with a concise review of hypothesis testing in particle physics, covering p‑values, Z‑scores, and the profile likelihood ratio. It then introduces the basic statistical machinery: maximum‑likelihood estimation, the Fisher information matrix, and the curvature (covariance) matrix. For binned data the Poisson likelihood leads to the familiar –2 ln λ expression, which under Wilks’ theorem follows a χ² distribution for sufficiently large samples.

The Asimov data set is defined as a deterministic data set whose expected values equal the true model parameters, thereby yielding the exact maximum‑likelihood estimators. Using this construct, the Wald approximation is derived: –2 ln λ(μ) ≈ (μ – μ′)²/σ², where σ² is obtained from the inverse Fisher information evaluated on the Asimov data. The Tevatron test statistic q = –2 ln(L_{s+b}/L_b) is then expressed in terms of the Wald quantity, showing that q follows a Gaussian distribution with mean 1 – 2 μ σ² and variance 4 σ². This Gaussianity permits direct calculation of p‑values via the cumulative normal distribution, eliminating the need for toy‑Monte‑Carlo ensembles.

To assess practical performance, the authors generate pseudo‑datasets that mimic a realistic analysis: 1 000 000 events distributed over 1500 histogram bins, with an exponential background and a mirrored exponential signal. They process these datasets through Collie, which computes exact profile likelihoods for signal strength μ and for nuisance parameters representing systematic uncertainties. Six families of systematic variations are explored:

1. Background‑only rate uncertainties (0 %–50 % in 5 % steps).
2. Simultaneous signal and background rate uncertainties (5 %–45 % in 10 % steps), both correlated and uncorrelated.
3. Asymmetric Gaussian “flat” uncertainties.
4. Shape distortions of the background model.
5. Varying the number of histogram bins (from 100 to 2000).
6. Varying the total event count (10 k, 100 k, 1 M).

For each scenario the signal strength required to achieve a 95 % confidence level (CL) is determined both by Collie (the “true” value) and by the A‑WW analytic formulas. The ratio of the analytic prediction to the Collie result is reported. Across all tests the ratio remains between 0.94 and 0.99, indicating a systematic under‑estimation of at most 6 % and, importantly, no monotonic degradation with increasing systematic size. Two‑dimensional heat‑maps for the simultaneous signal‑background uncertainties show a flat landscape, confirming that the approximations are robust to correlated nuisance parameters. Even when the binning is coarsened or the total event count is reduced, the approximations stay within 1 σ of the full calculation, provided the sample remains sufficiently large for the Gaussian approximation to hold.

The authors conclude that the A‑WW approximation is reliable for typical high‑energy physics analyses involving large datasets and modest systematic effects. Its computational speed makes it attractive for large‑scale scans, limit setting, and discovery significance estimation. However, they caution that in regimes with very low event counts, extreme asymmetries in systematic uncertainties, or highly non‑Poissonian fluctuations, the analytic formulas may deviate beyond the observed 5 % level, and a full Monte‑Carlo treatment remains advisable.

Overall, the paper provides a thorough validation of the asymptotic methods, quantifies their domain of applicability, and offers practical guidance for analysts deciding when to employ fast analytic approximations versus more exact, but computationally demanding, techniques.