The reliability of the AIC method in Cosmological Model Selection

The Akaike information criterion (AIC) has been used as a statistical criterion to compare the appropriateness of different dark energy candidate models underlying a particular data set. Under suitable conditions, the AIC is an indirect estimate of the Kullback-Leibler divergence D(T//A) of a candidate model A with respect to the truth T. Thus, a dark energy model with a smaller AIC is ranked as a better model, since it has a smaller Kullback-Leibler discrepancy with T. In this paper, we explore the impact of statistical errors in estimating the AIC during model comparison. Using a parametric bootstrap technique, we study the distribution of AIC differences between a set of candidate models due to different realizations of noise in the data and show that the shape and spread of this distribution can be quite varied. We also study the rate of success of the AIC procedure for different values of a threshold parameter popularly used in the literature. For plausible choices of true dark energy models, our studies suggest that investigating such distributions of AIC differences in addition to the threshold is useful in correctly interpreting comparisons of dark energy models using the AIC technique.

💡 Research Summary

The paper investigates the reliability of the Akaike Information Criterion (AIC) when used to select among competing dark‑energy models based on Type Ia supernova (SNIa) data. AIC is an asymptotically unbiased estimator of the Kullback‑Leibler (KL) divergence between a candidate model and the true underlying process; consequently, the model with the smallest AIC is regarded as the most plausible. However, AIC itself is estimated from a finite data set, and its value – and especially the difference between two models (ΔA,B) – is subject to statistical fluctuations that can be substantial when the number of observations is modest.

To quantify these fluctuations, the authors employ a parametric bootstrap approach. They select a reference model C (one of the candidate cosmologies) to generate synthetic data sets by resampling residuals under the assumption of Gaussian errors. For each bootstrap realization they refit the four cosmological models considered – ΛCDM (1 free parameter), wCDM (2 parameters), CPL (3 parameters), and DGP (1 parameter) – compute the maximum‑likelihood χ², and from it the AIC values (AIC = 2k + χ²). The resulting distribution of ΔA,B for each model pair is examined.

The bootstrap results reveal that the shape and spread of the Δ‑distributions differ markedly among model pairs. Some distributions are approximately Gaussian, while others are skewed, heavy‑tailed, or even multimodal. This variability implies that a single deterministic threshold (Δthreshold) for declaring one model “significantly better” than another is insufficient. The paper discusses the commonly used Δthreshold = 5 (suggested by Liddle 2007) and shows that, for the current SNIa sample (≈300 points), many model pairs fall within the indeterminate region (|Δ| < 5), leading to ambiguous conclusions. Lowering the threshold (e.g., Δ = 2) increases the fraction of decisive comparisons but also raises the risk of selecting the wrong model because the statistical noise can produce apparent large Δ values even when the true models are similar.

To provide a more nuanced decision metric, the authors define a “success rate”: the probability that the true model is identified as the best model given a particular Δthreshold. This success rate is estimated directly from the bootstrap Δ‑distributions. The analysis shows that the success rate improves as the data set grows, because the variance of Δ shrinks, making the traditional Δthreshold more reliable for larger samples. Nonetheless, for present‑day data the authors advocate reporting the full Δ‑distribution (or at least its mean, variance, and confidence intervals) alongside any threshold‑based decision, thereby giving a probabilistic interpretation of model selection.

The paper also briefly compares AIC with other information criteria such as BIC and Minimum Description Length, noting that while BIC penalizes model complexity more strongly, it suffers from similar statistical uncertainties. The authors conclude that, in cosmological model selection, the AIC should not be treated as a single scalar ranking but rather as a random variable whose distribution must be quantified. Incorporating bootstrap‑derived uncertainties allows researchers to assess the robustness of their conclusions, avoid over‑interpretation of marginal Δ values, and plan future surveys (DES, BIGBOSS, LSST, EUCLID) with a clearer understanding of the statistical power required to discriminate among dark‑energy scenarios.