Managing Many Simultaneous Systematic Uncertainties

Recent statistical evaluations for High-Energy Physics measurements, in particular those at the Large Hadron Collider, require careful evaluation of many sources of systematic uncertainties at the same time. While the fundamental aspects of the statistical treatment are now consolidated, both using a frequentist or a Bayesian approach, the management of many sources of uncertainties and their corresponding nuisance parameters in analyses that combine multiple control regions and decay channels, in practice, may pose challenging implementation issues, that make the analysis infrastructure complex and hard to manage, eventually resulting in simplifications in the treatment of systematics, and in limitations to the result interpretation. Typical cases will be discussed, having in mind the most popular implementation tool, RooStats, with possible ideas about improving the management of such cases in future software implementations.

💡 Research Summary

The paper addresses a practical and increasingly critical problem in modern high‑energy‑physics (HEP) data analysis: the simultaneous treatment of a large number of systematic uncertainties (often called “systematics”) together with the parameters of interest, typically a signal strength μ. While the underlying statistical foundations—both Bayesian marginalisation and frequentist profiling—are well established, the authors argue that the real bottleneck lies in the implementation of these methods in large‑scale LHC analyses that combine many control regions, decay channels, and data‑taking periods.

The authors begin by formalising the problem. A likelihood L(x; μ, θ) depends on the observable data x, the signal strength μ, and a vector of nuisance parameters θ that encode systematic effects. In a Bayesian framework the posterior P(μ, θ|x) ∝ L π(μ, θ) is obtained, and marginalisation over θ yields the posterior for μ. In the frequentist approach the profile likelihood λ(μ)=L(μ, θ̂̂)/L(μ̂, θ̂) is used; the test statistic qμ=−2 ln λ(μ) follows, under Wilks’ theorem, an asymptotic χ² distribution for a single parameter of interest, which enables rapid limit setting without extensive toy Monte‑Carlo.

A key practical extension is the inclusion of auxiliary (control) samples y that constrain θ. The joint likelihood becomes L(x,y; μ, θ)=Lx·Ly, where Ly may be independent of μ if the control region is signal‑free. When real control data are unavailable, the authors model Ly as a prior on the “nominal” values θnom (Gaussian, log‑normal, Gamma, etc.), leading to a likelihood term Lθnom(θnom; θ). This formulation captures the effect of external calibrations, test‑beam measurements, or theoretical priors.

Implementation is examined in the context of the RooFit/RooStats C++ framework, the de‑facto standard in ATLAS and CMS. Systematics are typically encoded as histogram templates: for each source of uncertainty a pair of ±1σ histograms is produced from simulation, and intermediate values are obtained by linear or quadratic interpolation (or extrapolation). The authors identify several pain points:

1. Naming and bookkeeping – Histograms must encode process, channel, systematic name, and up/down variation, often using underscores. In complex analyses the sheer number of files and overlapping names makes automated matching error‑prone.
2. Parameter explosion – When each bin’s statistical uncertainty is modelled with a separate nuisance parameter, the model can contain thousands of θ’s, inflating memory usage and minimisation time. Selecting only low‑populated bins for explicit treatment can mitigate this.
3. Formula handling – RooFormulaVar allows arbitrary algebraic expressions, but they are supplied as strings; syntax errors surface only at runtime, and complex scaling (e.g., h_sig_i = h_bkg_i α_i) becomes hard to read and debug.
4. Data‑card dependence – The CMS “combine” tool uses a datacard language to declare channels, processes, and systematics. While powerful, large analyses require ad‑hoc scripts to generate these cards, adding another maintenance layer.

To alleviate these issues, the paper proposes a hierarchical grouping of nuisance parameters: (i) global parameters common to all spectra (luminosity, jet‑energy scale, b‑tag efficiency); (ii) process‑specific theory uncertainties (renormalisation/factorisation scales, PDF variations); (iii) channel‑specific detector efficiencies (lepton ID, trigger); and (iv) spectrum‑specific statistical uncertainties. Such grouping can be expressed directly in the datacard syntax, reducing redundancy and simplifying the likelihood construction. The authors also suggest extending the CMS combine interface (or RooFit itself) with custom PDF classes (e.g., a “ScaledHistogramPdf”) that embed bin‑by‑bin scale factors, thereby avoiding string‑based formulae.

The INSIGHTS ITN project is highlighted as a concrete effort to develop these ideas. Funded by the EU Horizon 2020 programme, INSIGHTS brings together early‑stage researchers from ten European institutions to work on modern statistical tools, including machine‑learning‑based uncertainty quantification, and to prototype software extensions for RooStats. The project aims to produce reusable, open‑source modules that could eventually be merged into the official ROOT distribution.

In conclusion, the authors stress that while the statistical theory for handling many systematics is mature, the current software ecosystem forces analysts to write ad‑hoc, error‑prone code. A unified, modular solution built on RooFit/RooStats—featuring standardized naming conventions, automatic datacard generation, hierarchical nuisance‑parameter handling, and optional ML‑driven uncertainty models—would greatly streamline LHC analyses, improve reproducibility, and enable more robust physics interpretations.