Simulated Annealing ABC with multiple summary statistics

Bayesian inference for stochastic models is often challenging because evaluating the likelihood function typically requires integrating over a large number of latent variables. However, if only few parameters need to be inferred, it can be more efficient to perform the inference based on a comparison of the observations with (a large number of) model simulations, in terms of only few summary statistics. In Machine Learning (ML), Simulation Based Inference (SBI) using neural density estimation is often considered superior to the traditional sampling-based approach known as Approximate Bayesian Computation (ABC). Here, we present a new set of ABC algorithms based on Simulated Annealing and demonstrate that they are competitive with ML approaches, whilst requiring much less hyper-parameter tuning. For the design of these sampling algorithms we draw intuition from non-equilibrium thermodynamics, where we associate each summary statistic with a state variable (energy) quantifying the distance to the observed value as well as a temperature that controls the degree to which the associated statistic contributes to the posterior. We derive an optimal annealing schedule on a Riemannian manifold of state variables based on a minimal entropy production principle. Our new algorithms generalize the established Simulated Annealing based ABC to multiple state variables and temperatures. In situations where the information-content is unevenly distributed among the summary statistics, this can greatly improve performance of the algorithm. Our method also allows monitoring the convergence of individual statistics, which is a great diagnostic tool in out-of-sample situations. We validate our approach on standard benchmark tasks from the SBI literature and a hard inference problem from solar physics and demonstrate that it is highly competitive with the state-of-the-art.

💡 Research Summary

This paper introduces a novel family of Approximate Bayesian Computation (ABC) algorithms that integrate concepts from non‑equilibrium thermodynamics and simulated annealing to handle multiple summary statistics more effectively. Traditional ABC methods rely on a single distance metric and a global tolerance that must be manually tuned; this can be problematic when the chosen summary statistics differ widely in scale or information content. The authors address these issues by (1) assigning each summary statistic its own “energy” derived from the cumulative distribution function (CDF) of its distance under the prior, thereby normalizing disparate scales, and (2) equipping each statistic with an individual inverse temperature (βₑᵢ).

The algorithm proceeds as follows. A population of particles is drawn from the joint prior over parameters and simulated data. For each particle, the distances ρᵢ between simulated and observed statistics are transformed into energies uᵢ via the prior CDF (Eq. 2). Proposals are generated by perturbing the parameters, re‑simulating data, and recomputing the energies. Acceptance follows a Metropolis‑Hastings rule that depends on the sum of βₑᵢ (uᵢ′ − uᵢ) across all statistics, together with the prior ratio. Crucially, the temperatures βₑᵢ are not fixed; they are adapted during the run according to a schedule derived from a minimal entropy‑production principle.

The theoretical contribution rests on treating the particle ensemble as a thermodynamic system that, at each instant, approximates a Gibbs‑like distribution π_β(u,θ) ∝ f(u,θ) exp(−∑βᵢ uᵢ). Assuming endoreversibility (the system remains close to equilibrium) and linear response (∂U ≈ L F), the authors formulate the entropy production rate as \dot S = ∑ \dot Uᵢ Fᵢ, where Fᵢ = βₑᵢ − βᵢ. By minimizing this rate for fixed initial and final energies, they derive geodesic trajectories on a Riemannian manifold of the energy variables. After a series of approximations (broad prior, wide proposal kernel, near‑uniform prior of the energies), they obtain an explicit expression for the Onsager matrix L, its inverse metric g, and ultimately a closed‑form update rule for the external temperatures (Eq. 19). The schedule ensures that each βₑᵢ follows the optimal path that balances rapid convergence of informative statistics with stability for less informative ones.

Two algorithmic variants are presented. “SABC‑multi” uses distinct βₑᵢ for each statistic, fully exploiting the derived schedule. “SABC‑single” forces all βₑᵢ to be equal, reverting to the original single‑temperature annealing while still employing per‑statistic energies. Both variants require only a prior sample to estimate the CDF‑based energy transformations; no additional hyper‑parameters beyond the number of population updates are needed.

Empirical evaluation covers four benchmark problems common in simulation‑based inference (SBI): the two‑moons toy model, a hyperboloid model, a mixture model, and a Gaussian mixture with distractor (uninformative) statistics. Performance is measured with the classifier‑two‑sample‑test (C2ST) metric; lower scores indicate a posterior closer to the true distribution. Across multiple random seeds, SABC‑multi consistently outperforms or matches state‑of‑the‑art SBI methods such as Sequential Monte Carlo ABC (SMC‑ABC), Bayesian Neural Ratio Estimation (BNRE), and Neural Posterior Estimation (NPE) especially when uninformative statistics are present. The single‑temperature version remains competitive in more balanced settings.

Beyond synthetic tests, the authors apply the methods to three real‑world problems: (i) a high‑dimensional epidemiological model, (ii) a stochastic neural model with intractable likelihood, and (iii) a solar‑physics inference task involving observed solar flare data. In all cases, the multi‑temperature SABC achieves posterior estimates comparable to the best ML‑based SBI approaches while requiring far less hyper‑parameter tuning and offering transparent diagnostics. Notably, the per‑statistic temperature trajectories provide a built‑in monitor of convergence; statistics that fail to converge can be identified, aiding model criticism and out‑of‑sample diagnostics.

In summary, the paper makes three key contributions: (1) a principled way to normalize heterogeneous summary statistics via CDF‑based energy mapping, (2) a thermodynamically optimal annealing schedule derived from minimal entropy production that automatically adapts individual tolerances, and (3) a practical, low‑tuning ABC framework that rivals modern neural density‑estimation methods on both synthetic benchmarks and challenging real data. The work bridges statistical inference, thermodynamics, and algorithmic design, offering a robust alternative for practitioners facing complex stochastic models where likelihoods are unavailable but simulations are feasible.