Why we (usually) dont have to worry about multiple comparisons

Applied researchers often find themselves making statistical inferences in settings that would seem to require multiple comparisons adjustments. We challenge the Type I error paradigm that underlies these corrections. Moreover we posit that the problem of multiple comparisons can disappear entirely when viewed from a hierarchical Bayesian perspective. We propose building multilevel models in the settings where multiple comparisons arise. Multilevel models perform partial pooling (shifting estimates toward each other), whereas classical procedures typically keep the centers of intervals stationary, adjusting for multiple comparisons by making the intervals wider (or, equivalently, adjusting the $p$-values corresponding to intervals of fixed width). Thus, multilevel models address the multiple comparisons problem and also yield more efficient estimates, especially in settings with low group-level variation, which is where multiple comparisons are a particular concern.

💡 Research Summary

The paper challenges the conventional wisdom that researchers must apply multiple‑comparison adjustments whenever they test several hypotheses simultaneously. The authors argue that the prevailing Type I error framework—controlling the family‑wise error rate (FWER) or the false discovery rate (FDR) by widening confidence intervals or lowering p‑values—is often mismatched with the actual inferential goals of applied work. Most researchers are interested in estimating effect sizes and quantifying their uncertainty, not merely in making a binary “significant/not significant” decision for each comparison.

From a Bayesian perspective, the need for an external correction disappears because the model itself incorporates the multiplicity. Hierarchical (multilevel) models treat the parameters for each group, condition, or outcome as draws from a common population distribution. This structure induces partial pooling: individual estimates are shrunk toward the overall mean in proportion to their sampling variance and the estimated between‑group variance. Consequently, groups with small sample sizes or high within‑group noise are automatically pulled toward the grand mean, reducing the chance of spurious extreme estimates. In contrast, classical corrections keep the point estimates fixed and compensate for multiplicity solely by inflating interval widths, which can lead to overly conservative inference and loss of power.

The authors illustrate these ideas with two empirical examples—school‑level test scores and hospital‑level treatment effects—and with a series of simulations. In every scenario, the hierarchical Bayesian approach yields lower mean‑squared error for the estimated effects, while maintaining a false‑positive rate comparable to or better than traditional Bonferroni or Benjamini‑Hochberg adjustments. When the true between‑group differences are small or null, the Bayesian model almost always shrinks the estimates close to zero, effectively eliminating false discoveries without the need for an explicit correction. When genuine differences exist, the model retains enough flexibility to detect them, especially when the between‑group variance is appreciable.

Beyond point estimation, the Bayesian framework provides full posterior distributions for each effect. Researchers can therefore answer questions such as “What is the probability that this effect exceeds zero?” or “What is the probability that the effect lies within a practically important range?” These probabilistic statements are more informative for decision‑making than a simple p‑value threshold.

The paper also acknowledges limitations. The performance of hierarchical models depends on sensible choices of hyper‑priors and on the adequacy of the assumed exchangeability among groups. Poorly specified priors or misspecified hierarchical structure can lead to over‑shrinkage, masking real differences. Model convergence diagnostics, posterior predictive checks, and sensitivity analyses are essential to ensure robust inference. Moreover, in settings where groups are truly heterogeneous and the between‑group variance is large, partial pooling may be less desirable, and researchers might prefer a more “no‑pooling” approach.

In sum, the authors propose a paradigm shift: rather than treating multiple comparisons as a problem that must be corrected after the fact, they suggest embedding the multiplicity within the statistical model itself via hierarchical Bayesian methods. This approach simultaneously addresses the multiple‑comparison issue, yields more efficient (lower‑variance) estimates, and furnishes richer inferential statements. For applied researchers dealing with many related parameters—especially when group‑level variation is modest—the paper makes a compelling case that explicit multiple‑comparison adjustments are often unnecessary, provided a well‑specified multilevel Bayesian model is employed.