Understanding statistics for biomedical research through the lens of replication

Clinicians and scientists have traditionally focussed on whether their findings will be replicated and are very familiar with the concept. The probability that a replication study yields an effect with the same sign, or the same statistical significance as an original study depends on the sum of the variances of the effect estimates. On this basis, when P equals 0.025 one-sided and the replication study has the same sample size and variance as the original study, the probability of achieving a one-sided P is less than or equal to 0.025 a second time is only about 0.283, consistent with currently observed modest replication rates. A higher replication probability would require a larger sample size than that derived from current single variance power calculations. However, if the replication study is based on an infinitely large sample size and thus has negligible variance then the probability that its estimated mean is same sign is 1 - P = 0.975. The reasoning is made clearer by changing continuous distributions to discretised scales and probability masses, thus avoiding ambiguity and improper flat priors. This perspective is consistent with Frequentist and Bayesian interpretations and also requires further reasoning when testing scientific hypotheses and making decisions.

💡 Research Summary

This paper, “Understanding statistics for biomedical research through the lens of replication,” addresses the pervasive confusion surrounding statistical interpretation and the replication crisis by reframing core concepts through the intuitive idea of study replication. The central argument is that the common focus on a single study’s P-value is misleading for assessing the reliability of findings, and that replication probability offers a more meaningful metric.

The author begins by highlighting the challenges in interpreting P-values and confidence intervals, and the persistent divide between Frequentist and Bayesian paradigms. To build an intuitive foundation, the paper uses a concrete example of a crossover randomized controlled trial on blood pressure. It shows how probabilities (e.g., the chance an individual patient benefited from treatment) can be estimated directly from observed frequencies or from the cumulative distribution function of a Gaussian, without immediately invoking Bayes’ rule.

A key technical innovation proposed is the “discretisation” of continuous Gaussian distributions into very narrow intervals (e.g., 0.01 mmHg bins). This mirrors how measurements are actually recorded in medicine and science (as intervals) and resolves mathematical ambiguities inherent in continuous models, particularly the problem of “improper” flat priors in Bayesian analysis. By assuming a finite range of possible values, the author defines a “uniform prior probability conditional on the sample space.” This allows for the symmetry where the probability distribution of the true parameter given the statistic is identical in form to the likelihood distribution of the statistic given the parameter. Within this clarified framework, a one-sided P-value (e.g., 0.025) is shown to be not only the probability of obtaining the observed result (or more extreme) under the null hypothesis but also the probability that the true effect lies beyond the null hypothesis (e.g., is zero or less) given the observed data. Consequently, 1-P (e.g., 0.975) represents the probability that the true effect is on the same side as the observed effect.

The paper then applies this logic to the critical issue of replication. Predicting the result of a replication study is modeled as a two-stage convolution process: first, estimating the distribution of possible “true” effects from the original study result; second, estimating the distribution of possible replication results for each of those true effects. The variance of this combined, convolved distribution is the sum of the variances of the original and replication study effect estimates. This leads to the paper’s pivotal finding: when P = 0.025 (one-sided) in an original study, the probability that a replication study with the same sample size (and thus same variance) will also achieve P ≤ 0.025 is only about 28.3%. This aligns with empirically observed low replication rates. The analysis demonstrates that to achieve a high probability of successful replication (e.g., 97.5%), the replication study would need a sample size so large that its variance is negligible—far larger than what conventional power calculations for a single study would dictate.

In conclusion, the paper provides a coherent and accessible reinterpretation of statistical evidence by leveraging the concept of replication. By discretising distributions and introducing a sample-space-conditional uniform prior, it bridges Frequentist and Bayesian interpretations and offers practical insights for study design, sample size planning, and the critical evaluation of scientific and clinical evidence.