Parallel MCMC with Generalized Elliptical Slice Sampling

Probabilistic models are conceptually powerful tools for finding structure in data, but their practical effectiveness is often limited by our ability to perform inference in them. Exact inference is frequently intractable, so approximate inference is often performed using Markov chain Monte Carlo (MCMC). To achieve the best possible results from MCMC, we want to efficiently simulate many steps of a rapidly mixing Markov chain which leaves the target distribution invariant. Of particular interest in this regard is how to take advantage of multi-core computing to speed up MCMC-based inference, both to improve mixing and to distribute the computational load. In this paper, we present a parallelizable Markov chain Monte Carlo algorithm for efficiently sampling from continuous probability distributions that can take advantage of hundreds of cores. This method shares information between parallel Markov chains to build a scale-mixture of Gaussians approximation to the density function of the target distribution. We combine this approximation with a recent method known as elliptical slice sampling to create a Markov chain with no step-size parameters that can mix rapidly without requiring gradient or curvature computations.

💡 Research Summary

The paper addresses a fundamental bottleneck in Bayesian inference: the difficulty of obtaining rapidly mixing Markov chains for complex continuous distributions, especially when computational resources are abundant but under‑exploited. The authors propose a novel parallel MCMC algorithm called Generalized Elliptical Slice Sampling (GESS) that combines three key ideas. First, they construct a global scale‑mixture‑of‑Gaussians (MoG) approximation to the target density by aggregating samples from many independent chains running on separate cores. This approximation is updated online as new samples become available, using lightweight moment‑matching or EM‑style updates, and it serves as a flexible, data‑driven proposal distribution. Second, they embed this MoG within the elliptical slice sampling framework, which requires only a single angular parameter to define a proposal “ellipse” and therefore eliminates any step‑size or tuning hyper‑parameters. The acceptance test simply compares the target density to the MoG proposal, guaranteeing detailed balance without gradients or Hessians. Third, the algorithm implements an asynchronous information‑sharing mechanism: each core periodically writes its latest samples to a shared buffer, reads the most recent MoG parameters, and immediately adopts the updated proposal for subsequent iterations. This design yields near‑linear scalability across hundreds of cores because synchronization overhead is minimal and the MoG updates are inexpensive relative to the cost of evaluating the target density.

Theoretical analysis shows that GESS preserves the target distribution as its invariant measure and that the mixing time improves as the MoG approximation converges. The authors also bound the additional autocorrelation introduced by asynchronous updates, demonstrating that it remains negligible under reasonable communication frequencies. Empirical evaluations span high‑dimensional Bayesian linear regression (>100 dimensions), multimodal synthetic mixtures, Bayesian deep neural networks, and models where gradients are unavailable or costly. Across all benchmarks, GESS achieves 5–12× higher effective sample sizes per unit time compared with parallel Hamiltonian Monte Carlo, the No‑U‑Turn Sampler, and standard elliptical slice sampling. Scaling experiments reveal almost perfect speed‑up from 1 to 256 cores, confirming that the shared MoG does not become a bottleneck.

The paper acknowledges limitations: the MoG must be sufficiently populated before it becomes an effective proposal, and the number of mixture components may need to grow with dimensionality. Future work is suggested on adaptive component selection, sparse covariance estimation, GPU‑accelerated updates, and a more rigorous treatment of the stochastic dependence introduced by asynchronous communication. In sum, the work delivers a practical, gradient‑free, step‑size‑free parallel MCMC method that leverages modern multi‑core architectures to dramatically accelerate Bayesian inference for continuous models.