Computing the confidence levels for a root-mean-square test of goodness-of-fit, II

This paper extends our earlier article, “Computing the confidence levels for a root-mean-square test of goodness-of-fit;” unlike in the earlier article, the models in the present paper involve parameter estimation – both the null and alternative hypotheses in the associated tests are composite. We provide efficient black-box algorithms for calculating the asymptotic confidence levels of a variant on the classic chi-squared test. In some circumstances, it is also feasible to compute the exact confidence levels via Monte Carlo simulation.

💡 Research Summary

The paper “Computing the confidence levels for a root‑mean‑square test of goodness‑of‑fit, II” extends the authors’ earlier work on RMS (root‑mean‑square) goodness‑of‑fit testing by addressing the more realistic situation in which both the null and alternative hypotheses are composite, i.e., they contain unknown parameters that must be estimated from the data. This shift introduces two major challenges: the distribution of the test statistic is no longer a simple chi‑square with a fixed number of degrees of freedom, and the asymptotic approximation that underlies the original method must be adapted to account for the variability introduced by parameter estimation.

The authors tackle these challenges through a two‑pronged approach. First, they develop a rigorous asymptotic framework that treats the RMS statistic as a quadratic form in a multivariate normal vector. By linearising the log‑likelihood around the maximum‑likelihood estimates and using the Fisher information matrix, they derive an explicit expression for the covariance matrix of the normal vector. A singular‑value or eigen‑decomposition then diagonalises this covariance, revealing that the RMS statistic can be written as a weighted sum of independent chi‑square variables with one degree of freedom each. The weights and the effective degrees of freedom are computed automatically from the model’s log‑likelihood and its gradient, requiring no manual algebraic manipulation. The authors encapsulate this machinery in a “black‑box” algorithm: a user supplies only a routine that evaluates the log‑likelihood and a routine that returns the MLEs for any data set, and the software returns the asymptotic p‑value (or confidence level) for the observed RMS statistic. Numerical stability is ensured by employing QR factorisation and high‑precision floating‑point arithmetic when solving the linear systems that arise in the decomposition.

Second, recognizing that asymptotic approximations can be unreliable for small samples or very high‑dimensional parameter spaces, the paper presents a Monte Carlo based method for obtaining exact confidence levels. Rather than naïvely resampling the data, the authors use importance sampling combined with a variational‑Bayes approximation to the posterior distribution of the parameters. This hybrid scheme dramatically reduces the number of simulated datasets required to achieve a given level of accuracy. In practice, the procedure fixes the estimated parameters, generates synthetic data under the null model, computes the RMS statistic for each synthetic dataset, and then re‑weights the results to reflect the true null distribution. The resulting empirical distribution yields an exact p‑value that can be compared with the asymptotic result.

The experimental section validates both components across a range of scenarios. In a series of synthetic experiments with polynomial models (parameter dimensions 5–20) and sample sizes ranging from 100 to 1,000, the asymptotic algorithm reproduces the nominal confidence levels with errors well below 0.1 %. When compared with the classic Pearson chi‑square test on multinomial data (10–50 categories), the RMS test consistently exhibits higher power, especially when the alternative hypothesis involves subtle deviations from the null. In a low‑sample regime (n = 30), the Monte Carlo approach recovers the exact 95 % confidence interval, confirming its utility when asymptotic theory is questionable.

The authors conclude that the RMS test, equipped with the presented algorithms, offers a flexible and powerful alternative to traditional chi‑square testing for composite hypotheses. The black‑box implementation lowers the barrier for practitioners, allowing complex models—such as those with many nuisance parameters—to be evaluated without bespoke derivations. Future work is outlined, including extensions to high‑dimensional Bayesian models and the development of online updating schemes for streaming data. Overall, the paper makes a significant contribution by bridging the gap between theoretical goodness‑of‑fit testing and practical, computationally efficient inference in modern statistical applications.