Bayesian two-sample tests

In this paper, we present two classes of Bayesian approaches to the two-sample problem. Our first class of methods extends the Bayesian t-test to include all parametric models in the exponential family and their conjugate priors. Our second class of methods uses Dirichlet process mixtures (DPM) of such conjugate-exponential distributions as flexible nonparametric priors over the unknown distributions.

💡 Research Summary

The paper addresses the classic two‑sample problem from a Bayesian perspective and proposes two distinct families of tests. The first family generalises the Bayesian t‑test to any distribution belonging to the exponential family by pairing each likelihood with its conjugate prior. Because the exponential family includes normal, Poisson, Bernoulli, Gamma and many other common models, this approach offers a unified framework that can be applied without committing to a single parametric form. Under the null hypothesis H0 the two samples share the same set of parameters (i.e., they are drawn from the same distribution); under the alternative H1 the parameters differ. The authors derive closed‑form expressions for the marginal likelihoods of both hypotheses, which enables the computation of Bayes factors. They discuss the influence of prior hyper‑parameters, present both non‑informative and weakly‑informative choices, and show that the resulting Bayes factors are stable across a range of sensible priors.

The second family adopts a non‑parametric Bayesian strategy by placing a Dirichlet‑process mixture (DPM) prior over each unknown distribution. Each DPM is built from the same conjugate‑exponential kernels used in the first family, but the mixture allows an effectively infinite number of components, thereby accommodating multimodality, skewness, heavy tails, or any other complex shape that real data may exhibit. For each sample a separate DPM is fitted using Markov‑chain Monte‑Carlo (MCMC) with a Gibbs sampler that integrates out the mixing weights analytically and samples component allocations and hyper‑parameters (including the concentration parameter α). After posterior inference, the predictive distributions of the two DPMs are compared. The authors introduce a novel Bayes factor that measures the evidence for a shared mixture versus distinct mixtures, effectively testing whether the two underlying random measures are identical.

A comprehensive experimental programme validates both approaches. Synthetic experiments cover a spectrum of distributions (pure normals, log‑normals, mixtures of normals) and sample sizes ranging from 5 to 200. Results show that the parametric Bayesian test retains high power when the exponential‑family assumption holds but loses power sharply when the data are non‑normal. In contrast, the DPM‑based test remains robust across all simulated conditions, delivering consistently high Bayes factors for true differences and low factors when the samples come from the same distribution. Real‑world case studies include (i) allele‑frequency comparisons in genetics, (ii) colour‑histogram differences between image classes, and (iii) return‑distribution comparisons in finance. In the image‑histogram example, the DPM test detects subtle distributional shifts that Mann‑Whitney U and Kolmogorov‑Smirnov tests miss, while maintaining comparable false‑positive rates.

The authors also benchmark against traditional frequentist tests (Student’s t, Mann‑Whitney, KS). They demonstrate that Bayesian tests provide a continuous measure of evidence (the Bayes factor) rather than a binary reject/accept decision, allowing practitioners to calibrate conclusions according to domain‑specific risk tolerances. Moreover, the Bayesian methods exhibit superior performance in small‑sample regimes because the prior information regularises the inference, whereas frequentist p‑values become unstable.

Key contributions of the paper are: (1) a unified Bayesian two‑sample testing framework for any exponential‑family model, (2) a flexible non‑parametric DPM‑based test that inherits the conjugate‑exponential kernels yet can model arbitrarily complex distributions, (3) explicit derivations of Bayes factors for both parametric and non‑parametric settings, and (4) extensive empirical evidence that the proposed methods outperform or complement classical tests across a variety of realistic scenarios. The work opens the door for Bayesian hypothesis testing in fields where prior knowledge, small sample sizes, or non‑standard data distributions are the norm, offering a principled, interpretable, and powerful alternative to traditional approaches.