On Statistical Significance of Signal

A definition for the statistical significance of a signal in an experiment is proposed by establishing a correlation between the observed p-value and the normal distribution integral probability, which is suitable for both counting experiment and continuous test statistics. The explicit expressions to calculate the statistical significance for both cases are given.

💡 Research Summary

The paper addresses a long‑standing ambiguity in the way experimental physicists and astronomers translate a p‑value into the more intuitive “sigma” (σ) significance often quoted in discovery claims. While the p‑value is the fundamental probability of obtaining data at least as extreme as observed under the null hypothesis, the conversion to σ has traditionally relied on approximations that are not uniformly valid, especially for low‑count (Poisson) experiments or for test statistics that are not normally distributed.

The authors propose a unified definition that directly links the observed p‑value to the cumulative distribution function (CDF) of a standard normal distribution. Specifically, they set

 p = ∫ₛ^∞ f(x) dx = 1 – Φ(z),

where Φ(z) is the CDF of N(0, 1). Solving for z yields

 z = Φ⁻¹(1 – p).

In this formulation, z is taken as the statistical significance, i.e., the number of standard deviations that correspond to the tail probability p. This definition is inherently one‑sided, matching the usual practice in particle‑physics discovery criteria, but can be adapted to two‑sided tests by halving the tail probability.

The paper then derives explicit expressions for two common experimental contexts.

1. Counting experiments – The observed count n is modeled as a Poisson variable with expected background μ_b. The p‑value is

 p = Σ_{k=n}^{∞} e^{–μ_b} μ_b^k / k!.

Using the exact Poisson tail or a normal approximation (via the central limit theorem) the authors show how to compute z = Φ⁻¹(1 – p) without resorting to the ad‑hoc “σ ≈ √(2) erfc⁻¹(2p)” formula that is often used.

2. Continuous test statistics – For a generic statistic t with probability density f(t), the p‑value is defined as

 p = ∫_{t_obs}^{∞} f(t) dt.

Again, the significance is obtained by z = Φ⁻¹(1 – p). The authors emphasize that this works irrespective of whether f(t) is Gaussian; the only requirement is an accurate numerical evaluation of the tail integral.

To validate the approach, extensive Monte‑Carlo simulations are performed across a range of background levels and signal strengths. The results demonstrate that the proposed conversion yields smaller systematic biases than traditional approximations, particularly in the low‑count regime where Poisson fluctuations dominate. For continuous statistics, the method provides a consistent σ value even when the underlying distribution is skewed or has heavy tails.

Beyond the technical derivations, the authors discuss the broader impact of a standardized σ definition. Because σ is a language familiar to both specialists and the public (e.g., “5σ discovery”), tying it rigorously to the p‑value enhances the transparency and comparability of results across experiments. It also facilitates the communication of statistical confidence to policymakers and interdisciplinary audiences.

The paper concludes by suggesting extensions: incorporation of systematic uncertainties, multi‑parameter likelihood ratios, and connections to Bayesian posterior probabilities. The authors argue that their framework offers a robust, universally applicable tool for reporting statistical significance in modern experimental science.