Independent evaluation of the significance of the recent ATLAS and CMS data

This note describes an independent assessment of the statistical significance of the recently released ATLAS and CMS data, about 11 fb-1 per experiment acquired in 2011 and in the first part of 2012, for what concerns the Higgs search in the two high resolution decay channels especially suited for the low mass region, i.e. the diphoton and four-lepton decay channels. Scope of this note is not to reproduce the analysis of the Collaborations: this would be impossible given the enormous complexity of the complete profile likelihood procedure used to evaluate local and global the p-values, and the huge number of nuisance parameters which are used to incorporate the numerous systematic effects. Rather, its purpose is to show the significance that an outsider can infer only on the basis of the released data and plots, used as input of a simplified profile likelihood procedure in which the only contemplated nuisance parameter is the background normalization in the diphoton channel. In practice, this note tries to address the question of the independent judgment of the significance of new data that physicists are used to perform on their own when they are shown for the first time particularly relevant results with indication of new effects, and that in the complex LHC framework is not so easily doable as in other experimental contexts.

💡 Research Summary

The paper presents an independent, “outside‑the‑collaboration” assessment of the statistical significance of the Higgs‑boson search results released by the ATLAS and CMS experiments in 2011‑2012. Both experiments each collected roughly 11 fb⁻¹ of proton‑proton collisions at √s = 7 TeV, and the author focuses on the two high‑resolution decay channels that dominate the low‑mass region: the diphoton (γγ) and the four‑lepton (4ℓ) final states.

Because the full ATLAS and CMS analyses rely on a sophisticated profile‑likelihood framework that incorporates hundreds of nuisance parameters (NPs) to model systematic uncertainties (detector calibration, background shape, theoretical cross‑sections, etc.), an outsider cannot faithfully reproduce the official results. Instead, the author adopts a deliberately simplified approach that uses only the publicly released histograms, peak positions, and event counts. The only nuisance parameter retained is the overall background normalisation in the diphoton channel; all other systematic effects are ignored.

Methodologically, the author builds a minimal signal‑plus‑background model for each channel. In the γγ channel the signal is modeled as a Gaussian peak superimposed on a smoothly varying background (parameterised by a low‑order polynomial or exponential). In the 4ℓ channel a Breit‑Wigner shape is used for the signal, again on top of a simple background. The signal strength μ (the ratio of the observed signal yield to the Standard Model expectation) and the background normalisation β are the only free parameters. A profile likelihood ratio λ(μ) is constructed, and the test statistic q₀ = −2 ln λ(μ = 0) is evaluated. Under the asymptotic χ²(1) approximation, q₀ yields a local p‑value, which is then converted into a significance Z (in units of σ). To obtain a global significance the author applies a “look‑elsewhere” correction that accounts for the fact that the search was performed over a mass interval (roughly 110–150 GeV).

The simplified analysis finds a local excess of about 3 σ in the diphoton spectrum near 124 GeV and an excess of roughly 2 σ in the four‑lepton spectrum around 125 GeV. After the global correction, both channels fall below 2 σ. These numbers are considerably smaller than the ≈5 σ combined significance reported by the collaborations. The author attributes the discrepancy to two main factors: (1) the oversimplified background model can artificially inflate the apparent signal‑to‑background ratio, and (2) the omission of the many systematic nuisance parameters dramatically reduces the total uncertainty, leading to an under‑estimation of the true p‑value.

Despite these limitations, the paper demonstrates that a reasonable, albeit coarse, estimate of significance can be obtained from publicly available information. This “first‑look” independent check serves several important purposes. It illustrates that the existence of a local excess is robust enough to be seen even with a stripped‑down statistical treatment, while also emphasizing the necessity of a proper global correction. Moreover, it highlights the value of transparency: when experiments release enough detail (histograms, background estimates, systematic breakdowns), external researchers can perform sanity checks, fostering confidence in the results and encouraging a culture of reproducibility.

The author concludes by urging the LHC collaborations to continue and expand the practice of data sharing, perhaps by providing simplified likelihoods or RooFit workspaces that encapsulate the full systematic model. Such resources would enable the community to conduct more refined independent analyses, to cross‑validate the official findings, and ultimately to strengthen the credibility of discoveries in high‑energy physics. The paper thus serves both as a methodological case study and as a call for greater openness in large‑scale experimental collaborations.