Asymptotically Exact, Embarrassingly Parallel MCMC

Communication costs, resulting from synchronization requirements during learning, can greatly slow down many parallel machine learning algorithms. In this paper, we present a parallel Markov chain Monte Carlo (MCMC) algorithm in which subsets of data are processed independently, with very little communication. First, we arbitrarily partition data onto multiple machines. Then, on each machine, any classical MCMC method (e.g., Gibbs sampling) may be used to draw samples from a posterior distribution given the data subset. Finally, the samples from each machine are combined to form samples from the full posterior. This embarrassingly parallel algorithm allows each machine to act independently on a subset of the data (without communication) until the final combination stage. We prove that our algorithm generates asymptotically exact samples and empirically demonstrate its ability to parallelize burn-in and sampling in several models.

💡 Research Summary

The paper addresses a fundamental bottleneck in large‑scale Bayesian inference: the need for frequent communication among distributed workers when running Markov chain Monte Carlo (MCMC). Traditional parallel MCMC either runs many independent chains on the full dataset (which does not reduce burn‑in time) or requires each iteration to exchange information across machines (which incurs heavy synchronization costs). The authors propose an “embarrassingly parallel” MCMC framework that eliminates inter‑machine communication during the sampling phase and requires communication only once, after all local chains have finished.

The method proceeds in two stages. First, the full data set (x_{1:N}) is arbitrarily partitioned into (M) disjoint subsets ({x^{(1)},\dots,x^{(M)}}). For each subset a sub‑posterior density is defined as
\