Quantitative model validation techniques: new insights

This paper develops new insights into quantitative methods for the validation of computational model prediction. Four types of methods are investigated, namely classical and Bayesian hypothesis testing, a reliability-based method, and an area metric-based method. Traditional Bayesian hypothesis testing is extended based on interval hypotheses on distribution parameters and equality hypotheses on probability distributions, in order to validate models with deterministic/stochastic output for given inputs. Two types of validation experiments are considered - fully characterized (all the model/experimental inputs are measured and reported as point values) and partially characterized (some of the model/experimental inputs are not measured or are reported as intervals). Bayesian hypothesis testing can minimize the risk in model selection by properly choosing the model acceptance threshold, and its results can be used in model averaging to avoid Type I/II errors. It is shown that Bayesian interval hypothesis testing, the reliability-based method, and the area metric-based method can account for the existence of directional bias, where the mean predictions of a numerical model may be consistently below or above the corresponding experimental observations. It is also found that under some specific conditions, the Bayes factor metric in Bayesian equality hypothesis testing and the reliability-based metric can both be mathematically related to the p-value metric in classical hypothesis testing. Numerical studies are conducted to apply the above validation methods to gas damping prediction for radio frequency (RF) microelectromechanical system (MEMS) switches. The model of interest is a general polynomial chaos (gPC) surrogate model constructed based on expensive runs of a physics-based simulation model, and validation data are collected from fully characterized experiments.

💡 Research Summary

This paper presents a comprehensive study of quantitative techniques for validating computational model predictions, focusing on four distinct approaches: classical hypothesis testing, Bayesian hypothesis testing, a reliability‑based method, and an area‑metric method. The authors first lay out the mathematical foundations of each technique and then discuss how they can be applied under two experimental regimes—fully characterized experiments, where every input to the model and the corresponding experiment is measured as a point value, and partially characterized experiments, where some inputs are either unmeasured or reported as intervals.

In the classical framework, a null hypothesis (the model accurately represents the physical system) is tested against an alternative using a test statistic and the associated p‑value. The decision rule follows the conventional significance level (e.g., α = 0.05). While straightforward, this approach does not directly incorporate prior knowledge or the cost of Type I/II errors.

The Bayesian framework extends traditional Bayes‑factor analysis in two novel directions. First, interval hypotheses are introduced, allowing the analyst to test whether model parameters lie within prescribed bounds rather than being equal to a single value. Second, equality hypotheses are defined on entire probability distributions, enabling validation of models that produce stochastic outputs. By selecting an appropriate acceptance threshold for the Bayes factor, the analyst can explicitly control the risk of erroneous model selection. Moreover, the Bayes‑factor results can be fed into Bayesian model averaging, thereby mitigating the impact of a single erroneous decision. The paper demonstrates that, under assumptions of normality and equal variance, the Bayes factor can be mathematically related to the classical p‑value, providing a bridge between the two paradigms.

The reliability‑based method quantifies the probability that a model prediction falls within a user‑specified tolerance of the experimental observation. This probability, often called “reliability,” is obtained by propagating input uncertainties through the model (using, for example, polynomial chaos expansions) and then integrating the resulting predictive distribution over the tolerance interval. The method naturally accommodates directional bias: if the model mean is systematically higher (or lower) than the data, the reliability will be reduced, and the bias magnitude can be extracted from the reliability curve.

The area‑metric method evaluates the discrepancy between the cumulative distribution functions (CDFs) of model predictions and experimental data. By integrating the absolute difference between the two CDFs, an “area metric” is obtained; smaller areas indicate better agreement. The sign of the area (positive or negative) reveals whether the model tends to over‑predict or under‑predict, thus providing a simple visual and quantitative measure of directional bias.

A key contribution of the work is the systematic treatment of partially characterized experiments. When some inputs are reported only as intervals, the authors show how each validation technique can be adapted: Bayesian interval hypotheses treat the unknown inputs as additional random variables with uniform priors over the reported intervals; the reliability method expands the tolerance region to include input uncertainty; and the area metric integrates over the range of possible CDFs generated by the interval inputs.

The methodological developments are illustrated on a realistic engineering problem: gas‑damping prediction for radio‑frequency (RF) micro‑electromechanical system (MEMS) switches. A high‑fidelity physics‑based simulation is used to generate training data for a generalized polynomial chaos (gPC) surrogate model. Validation data are obtained from fully characterized laboratory experiments. Applying the four techniques yields consistent findings: the classical test rejects the surrogate at the 5 % level, whereas the Bayesian interval hypothesis, the reliability metric (≈ 0.78 probability of meeting tolerance), and the area metric (small positive area) all indicate a modest but systematic under‑prediction bias. The Bayesian analysis further demonstrates how adjusting the prior or the loss function can either retain the model for downstream use or down‑weight it in a model‑averaging ensemble.

Overall, the paper argues that relying on a single validation metric is risky. By employing a suite of complementary methods, engineers can detect and quantify bias, incorporate input uncertainty, and make informed decisions about model acceptance, revision, or replacement. The demonstrated mathematical link between Bayes factors and p‑values, together with the practical guidelines for handling interval‑type inputs, equips practitioners with a robust, unified framework for model validation across a wide range of scientific and engineering applications.