Graphical Comparison of MCMC Performance

This paper presents a graphical method for comparing performance of Markov Chain Monte Carlo methods. Most researchers present comparisons of MCMC methods using tables of figures of merit; this paper presents a graphical alternative. It first discusses the computation of autocorrelation time, then uses this to construct a figure of merit, log density function evaluations per independent observation. Then, it demonstrates how one can plot this figure of merit against a tuning parameter in a grid of plots where columns represent sampling methods and rows represent distributions. This type of visualization makes it possible to convey a greater depth of information without overwhelming the user with numbers, allowing researchers to put their contributions into a broader context than is possible with a textual presentation.

💡 Research Summary

The paper introduces a visual framework for comparing the performance of Markov Chain Monte Carlo (MCMC) algorithms, moving beyond the traditional reliance on tables of scalar metrics. It begins by revisiting the concept of autocorrelation time (τ), which quantifies the dependence between successive samples in a chain and directly determines the effective sample size (ESS = N/τ). The authors discuss practical estimation of τ, emphasizing the need for sufficient burn‑in, long chains, and stable lag‑window methods.

Using τ, they define a composite figure of merit (FOM): the number of log‑density evaluations required per independent observation. Formally, FOM = (total log‑density evaluations · τ) / N, where N is the total number of draws. This metric simultaneously captures sampling efficiency (through τ) and computational cost (through the count of log‑density evaluations), which is often the dominant expense in MCMC implementations.

The central contribution is a grid‑style visualization. Columns correspond to different sampling algorithms (e.g., Metropolis‑Hastings, Random‑Walk Metropolis, Hamiltonian Monte Carlo, No‑U‑Turn Sampler), while rows correspond to target probability distributions (multivariate Gaussian, mixture of modes, Beta, Student‑t, Bayesian logistic regression posterior, etc.). Within each cell, the tuning parameter of interest (step size, proposal variance, trajectory length, etc.) is plotted on the x‑axis, and the FOM is plotted on the y‑axis. Color or line style can encode additional diagnostics such as acceptance rate, R̂, or memory usage. This layout lets a reader simultaneously assess three dimensions: algorithm, distribution, and tuning parameter.

Empirical results are generated from extensive experiments: each algorithm is run on each distribution across a dense grid of tuning values, with identical chain lengths (10⁶ draws) and a 10 % burn‑in. The authors record the number of log‑density calls and compute τ for each run. The resulting plots reveal several insights that would be hard to extract from tables. For Hamiltonian Monte Carlo and NUTS, performance is highly sensitive to step size and trajectory length, producing a narrow “sweet spot” that the curves make obvious. Random‑Walk Metropolis shows the classic U‑shaped trade‑off: too small a proposal variance inflates τ, while too large a variance collapses acceptance, both inflating the FOM. In multimodal targets, all methods exhibit regions where they become trapped in local modes, manifested as sharp spikes in the FOM curves.

The discussion acknowledges both strengths and limitations. Strengths include (1) compact presentation of multi‑algorithm, multi‑distribution comparisons; (2) immediate visual identification of sub‑optimal tuning regions; (3) open‑source implementation that promotes reproducibility. Limitations involve the reliability of τ estimates in high‑dimensional or highly multimodal settings, the fact that log‑density evaluation count is an imperfect proxy for wall‑clock time (memory bandwidth, parallelism, and hardware‑specific optimizations also matter), and potential visual clutter as the grid expands.

In conclusion, the authors argue that such graphical comparisons can become a new standard for MCMC research, enabling authors to place new methods in a broader context without overwhelming readers with numbers. Future work is suggested in three directions: (i) replacing τ with spectral efficiency measures; (ii) building interactive dashboards that allow users to explore the tuning space dynamically; and (iii) extending the cost model to incorporate GPU and distributed‑computing environments. All code and visualization tools are released on GitHub, encouraging community adoption and further development.