Competitive Comparison of Optimal Designs of Experiments for Sampling-based Sensitivity Analysis

Nowadays, the numerical models of real-world structures are more precise, more complex and, of course, more time-consuming. Despite the growth of a computational effort, the exploration of model behaviour remains a complex task. The sensitivity analysis is a basic tool for investigating the sensitivity of the model to its inputs. One widely used strategy to assess the sensitivity is based on a finite set of simulations for a given sets of input parameters, i.e. points in the design space. An estimate of the sensitivity can be then obtained by computing correlations between the input parameters and the chosen response of the model. The accuracy of the sensitivity prediction depends on the choice of design points called the design of experiments. The aim of the presented paper is to review and compare available criteria determining the quality of the design of experiments suitable for sampling-based sensitivity analysis.

💡 Research Summary

The paper addresses a fundamental challenge in modern computational engineering: performing sensitivity analysis on increasingly complex and time‑consuming numerical models. Because direct derivative‑based methods are often infeasible for high‑fidelity, multi‑physics simulations, practitioners frequently resort to sampling‑based sensitivity analysis, where a finite set of input configurations (design points) is simulated and statistical correlations between inputs and outputs are computed. The accuracy of the resulting sensitivity indices, however, hinges critically on the quality of the experimental design (DoE) used to generate those points.

The authors first review the most widely adopted criteria for evaluating DoE quality in the context of sampling‑based sensitivity analysis. Four major families are identified: (1) Space‑filling criteria, which aim to distribute points uniformly throughout the admissible input domain. Representative measures include the Maximin distance (maximizing the smallest inter‑point distance), Minimax distance, average distance, and discrepancy metrics (L2, L∞). These criteria are particularly important for avoiding clustering and ensuring that all regions of the input space are explored. (2) Orthogonality (or regularity) criteria, which focus on minimizing statistical dependence among the columns of the design matrix. Common diagnostics are the absolute sum of the correlation matrix, the condition number of the matrix, and principal‑component‑based independence ratios. High orthogonality reduces multicollinearity, leading to more reliable regression‑based sensitivity estimates. (3) D‑optimality, a classical statistical optimality concept that maximizes the determinant of the information matrix (XᵀX). By maximizing this determinant, D‑optimal designs minimize the variance of estimated regression coefficients for a given number of runs, making them attractive when simulation budgets are tight. (4) Composite or multi‑objective criteria, which combine the previous objectives into a single optimization problem. The authors illustrate the use of evolutionary multi‑objective algorithms (e.g., NSGA‑II) to simultaneously improve space‑filling, orthogonality, and D‑optimality.

To assess the practical impact of each criterion, the study conducts a systematic numerical experiment campaign. First, low‑dimensional benchmark functions (Rosenbrock, Ackley, etc.) are used to illustrate basic behavior. Then, higher‑dimensional test cases (3, 5, and 10 input variables) are examined using a suite of designs: simple random sampling, Latin Hypercube Sampling (LHS), Maximin‑LHS, Sobol quasi‑random sequences, Halton sequences, D‑optimal designs generated via exchange algorithms, and the composite designs obtained from the multi‑objective optimizer. For each design, the authors run a Monte‑Carlo simulation with 1,000 replications, compute first‑order and total Sobol indices analytically (or via a high‑accuracy reference), and compare them to the indices estimated from the sampled data. Performance metrics include mean absolute error (MAE), maximum absolute error, and the coefficient of determination (R²) between estimated and reference indices.

The results reveal nuanced trade‑offs. Space‑filling designs (Maximin‑LHS, Sobol) consistently achieve the lowest average errors across all dimensions, confirming that a uniform coverage of the input domain is essential for accurate global sensitivity measures. However, when the design matrix exhibits strong column correlations, regression‑based estimators become unstable, leading to larger variance in the Sobol estimates. Orthogonal designs (low correlation, low condition number) mitigate this issue but may sacrifice uniform coverage, especially in high dimensions, resulting in higher bias for non‑linear effects. D‑optimal designs excel when the number of samples is severely limited (e.g., ≤ 50), delivering precise regression coefficients and consequently reliable first‑order Sobol estimates. Their main drawback is a tendency to concentrate points near the centre of the design space, which can under‑sample boundary regions where non‑linear responses often dominate. Composite designs generated by NSGA‑II strike a balance: they retain sufficient space‑filling to capture global trends, maintain acceptable orthogonality to keep regression variance low, and preserve a relatively large determinant of the information matrix. In dimensions of ten or more, these multi‑objective designs achieve the smallest MAE and MAE_max simultaneously, even with modest sample sizes (≈ 80–100).

Based on these findings, the authors propose a practical decision framework for selecting an appropriate DoE in sampling‑based sensitivity analysis:

1. Abundant computational budget (≥ 200 runs) – prioritize pure space‑filling designs such as Maximin‑LHS or Sobol sequences; they are easy to generate and provide robust global sensitivity estimates.
2. Tight budget with regression‑based analysis – adopt D‑optimal designs, but verify orthogonality post‑hoc (e.g., condition number < 10) and, if necessary, augment with a few additional space‑filling points to improve boundary coverage.
3. High‑dimensional, highly non‑linear models – employ multi‑objective composite designs; the evolutionary optimizer can be tuned to weight space‑filling more heavily for strongly non‑linear problems, or orthogonality for models with strong input interactions.
4. Quality assurance – regardless of the chosen method, compute diagnostic metrics (minimum inter‑point distance, condition number, determinant of XᵀX) after generation to confirm that the design meets the intended criteria.

In conclusion, the paper delivers a comprehensive, quantitative comparison of DoE quality criteria tailored to sampling‑based sensitivity analysis. By linking theoretical optimality concepts with concrete numerical experiments, it clarifies when each criterion is most beneficial and provides actionable guidelines for engineers and researchers tasked with extracting reliable sensitivity information from expensive, high‑fidelity simulations.