Data set diagonalization in a global fit

The analysis of data sometimes requires fitting many free parameters in a theory to a large number of data points. Questions naturally arise about the compatibility of specific subsets of the data, such as those from a particular experiment or those based on a particular technique, with the rest of the data. Questions also arise about which theory parameters are determined by specific subsets of the data. I present a method to answer both of these kinds of questions. The method is illustrated by applications to recent work on measuring parton distribution functions.

💡 Research Summary

The paper addresses a common challenge in global analyses that involve fitting a large number of theoretical parameters to extensive data sets: determining how specific subsets of the data (e.g., from a particular experiment or measurement technique) are compatible with the rest of the data and identifying which parameters are primarily constrained by those subsets. Traditional global fits minimize a total χ² and use the Hessian matrix to estimate parameter uncertainties and correlations, but they do not provide a clear, quantitative way to isolate the influence of individual data groups.

To solve this, the author introduces the Data Set Diagonalization (DSD) method. The procedure begins with the standard Hessian analysis of the full data set, yielding eigenvalues λ_i and eigenvectors v_i. By defining new coordinates x_i = √λ_i (v_i·δa), where δa denotes a small shift in the parameter vector, the total χ² becomes a simple sum Σ x_i², i.e., a sphere in the transformed space. The next step projects the χ² contribution of a chosen subset onto the same basis, producing a weighted sum Σ w_i x_i². The weights w_i quantify how much each eigen‑direction is represented in the subset. By orthogonalizing the weight vector, the method separates the parameter space into two independent directions: a “common” direction that is shared with the rest of the data, and a “specific” direction that is unique to the subset. In this new basis the total χ² and the subset χ² are each expressed as the sum of squares of two independent variables, allowing a direct comparison of the subset’s impact.

If the subset is fully compatible with the global fit, the specific direction contributes little to the total χ² and the parameter shifts remain within the expected uncertainties. Conversely, a large χ² contribution from the specific direction signals tension, and the associated parameter variations become amplified, revealing which parameters are driving the disagreement.

The method is demonstrated on modern parton distribution function (PDF) fits. Two representative subsets are examined: deep‑inelastic scattering data from HERA and high‑p_T jet data from the Tevatron and LHC. DSD shows that HERA data predominantly constrain the low‑x gluon distribution, while the jet data are most sensitive to high‑x quark PDFs. The χ² contributions from the specific directions of each subset are modest, indicating that the current PDF parametrization accommodates both data sets without significant conflict.

Key advantages of DSD are: (1) it provides a clear, visual decomposition of the parameter space into directions that are common or unique to a data subset; (2) it yields a quantitative test of compatibility based on the χ² of the specific direction; and (3) it integrates seamlessly with the standard Hessian error‑propagation framework, requiring only modest additional computation. The authors argue that DSD can be applied broadly to any global fitting problem in particle physics, astrophysics, or even machine‑learning model calibration, wherever large, heterogeneous data sets must be reconciled.

In summary, Data Set Diagonalization offers a powerful, mathematically rigorous tool for dissecting the influence of individual data groups within a global fit, clarifying parameter sensitivities, and diagnosing potential inconsistencies, thereby enhancing the reliability and interpretability of complex multi‑parameter analyses.