Assessing the properties of the prediction interval in random-effects meta-analysis

Random effects meta-analysis is a widely applied methodology to synthetize research findings of studies in a specific scientific question. Besides estimating the mean effect, an important aim of the meta-analysis is to summarize the heterogeneity, i.e. the variation in the underlying effects caused by the differences in study circumstances. The prediction interval is frequently used for this purpose: a 95% prediction interval contains the true effect of a similar new study in 95% of the cases when it is constructed, or in other words, it covers 95% of the true effects distribution on average. In this article, after providing a clear mathematical background, we present an extensive simulation investigating the performance of all frequentist prediction interval methods published to date. The work focuses on the distribution of the coverage probabilities and how these distributions change depending on the amount of heterogeneity and the number of involved studies. Although the single requirement that a prediction interval has to fulfill is to keep a nominal coverage probability on average, we demonstrate why the distribution of coverages cannot be disregarded, and that for small number of studies no reliable conclusion can be drawn from the prediction interval. We argue that assessing only the mean coverage can easily lead to misunderstanding and misinterpretation. The length of the intervals and the robustness of the methods concerning non-normality of the true effects are also investigated.

💡 Research Summary

This paper provides a comprehensive evaluation of frequentist prediction interval (PI) methods used in random‑effects meta‑analysis, focusing not only on the traditional metric of average coverage probability but also on the distributional properties of coverage, interval length, and robustness to violations of the normality assumption for the true effects distribution. The authors begin by outlining the random‑effects model, where study‑specific true effects are assumed to be independent draws from a common distribution with mean μ and variance τ², and observed outcomes are noisy estimates of these true effects. They discuss standard estimators for μ and τ², namely the DerSimonian‑Laird (DL) method of moments and restricted maximum likelihood (REML), and note that heterogeneity is typically quantified by τ², I², or the Q‑test.

The central object of interest, the prediction interval, is defined as a random interval (L(D), U(D)) constructed from the observed data D such that the expected coverage E