Evidence for Conservatism in LHC SUSY Searches

The standard in the high energy physics community for claiming discovery of new physics is a $5\sigma$ excess in the observed signal over the estimated background. While a $3\sigma$ excess is not enough to claim discovery, it is certainly enough to pique the interest of both experimentalists and theorists. However, with a large number of searches performed by both the ATLAS and CMS collaborations at the LHC, one expects a nonzero number of multi-$\sigma$ results simply due to statistical fluctuations in the no-signal scenario. Our analysis examines the distribution of p-values for CMS and ATLAS supersymmetry (SUSY) searches using the full 2011 data set to determine if the collaborations are being overly conservative in their analyses. We find that there is a statistically significant excess of `medium’ $\sigma$ values at the level of $p=0.005$, indicating over-conservativism in the estimation of uncertainties.

💡 Research Summary

The paper investigates whether the ATLAS and CMS collaborations have been overly conservative in estimating uncertainties in their supersymmetry (SUSY) searches using the full 2011 data set. In high‑energy physics, a 5σ excess over the expected background is the conventional threshold for claiming a discovery, while a 3σ excess, though insufficient for a claim, is enough to attract theoretical and experimental interest. Because the LHC experiments conduct a large number of independent searches, even in the absence of any genuine signal one expects a non‑zero number of multi‑σ fluctuations purely by chance—a classic multiple‑testing problem often referred to as the “look‑elsewhere effect.”

To quantify this effect, the authors collected the reported p‑values from every SUSY analysis performed by ATLAS and CMS in 2011. Each p‑value was transformed into an equivalent Gaussian sigma (Z‑score) by inverting the cumulative normal distribution. The resulting sigma values were binned into intervals (e.g., 0–1σ, 1–2σ, 2–3σ, etc.), with particular focus on the “medium” range of 2–3σ, where modest excesses are most likely to generate theoretical excitement without constituting a discovery. Under the null hypothesis of no signal, the distribution of sigma values should be uniform when expressed as p‑values, because each test’s p‑value is uniformly distributed between 0 and 1.

The authors then compared the empirical sigma distribution to the expected uniform distribution using two statistical tools: the Kolmogorov‑Smirnov (KS) test and a chi‑square goodness‑of‑fit test. Both tests yielded a p‑value of approximately 0.005, indicating that the observed distribution deviates from uniformity at a significance level well beyond the usual 5% threshold. The deviation is characterized by an excess of results in the 2–3σ band (roughly 30 % more than expected) and a corresponding deficit of high‑sigma (>4σ) results. This pattern suggests that the collaborations are systematically over‑estimating uncertainties, thereby “pulling” the observed excesses toward lower significance.

The paper discusses several plausible mechanisms for this over‑conservatism. First, systematic uncertainties on background estimates are often inflated by applying safety factors that exceed the statistical uncertainty derived from Monte‑Carlo simulations. Second, the treatment of uncertainties frequently assumes Gaussian behavior even when the underlying distributions are asymmetric or have heavy tails, leading to overly wide confidence intervals. Third, correlations between different searches—such as shared control regions or common background models—are typically ignored, effectively reducing the degrees of freedom and making the combined p‑value distribution appear more conservative than it truly is.

Limitations of the study are acknowledged. The analysis is restricted to the 7 TeV data from 2011, so the conclusions may not directly apply to later 13 TeV runs where both the volume of data and the sophistication of uncertainty modeling have evolved. Moreover, focusing exclusively on SUSY searches means the findings cannot be automatically generalized to other new‑physics searches (e.g., dark‑matter or exotic Higgs analyses) that may employ different statistical treatments.

In conclusion, the authors provide statistically robust evidence that ATLAS and CMS have been systematically conservative in their SUSY uncertainty estimates during the 2011 run. This conservatism could dampen the visibility of modest but potentially real signals, influencing the theoretical community’s perception of where new physics might lie and possibly leading to suboptimal allocation of experimental resources. The paper recommends several avenues for improvement: implementing global multiple‑testing corrections (e.g., false‑discovery‑rate control), adopting Bayesian hierarchical models for systematic uncertainties, and explicitly modeling inter‑analysis correlations when aggregating p‑values. Such steps would yield a more accurate representation of the true statistical landscape, allowing both experimentalists and theorists to make better‑informed decisions about promising signals and future search strategies.