Expectation Particle Belief Propagation

We propose an original particle-based implementation of the Loopy Belief Propagation (LPB) algorithm for pairwise Markov Random Fields (MRF) on a continuous state space. The algorithm constructs adaptively efficient proposal distributions approximating the local beliefs at each note of the MRF. This is achieved by considering proposal distributions in the exponential family whose parameters are updated iterately in an Expectation Propagation (EP) framework. The proposed particle scheme provides consistent estimation of the LBP marginals as the number of particles increases. We demonstrate that it provides more accurate results than the Particle Belief Propagation (PBP) algorithm of Ihler and McAllester (2009) at a fraction of the computational cost and is additionally more robust empirically. The computational complexity of our algorithm at each iteration is quadratic in the number of particles. We also propose an accelerated implementation with sub-quadratic computational complexity which still provides consistent estimates of the loopy BP marginal distributions and performs almost as well as the original procedure.

💡 Research Summary

The paper introduces a novel particle‑based algorithm for performing Loopy Belief Propagation (LBP) on pairwise Markov Random Fields (MRFs) with continuous, possibly unbounded, state spaces. Traditional non‑parametric approaches such as Nonparametric Belief Propagation (NBP) suffer from restrictive integrability conditions and lack of consistency, while Particle Belief Propagation (PBP) mitigates some of these issues but relies on costly and biased MCMC approximations of the proposal distribution (typically the current belief). The authors propose Expectation Particle Belief Propagation (EPBP), which retains the importance‑sampling spirit of PBP but replaces the intractable proposal with a tractable exponential‑family distribution whose parameters are updated iteratively using Expectation Propagation (EP).

In EPBP each node u maintains a proposal distribution q_u(x_u) that factorises as a product of exponential‑family factors approximating the node potential ψ_u and each incoming message b_m^{wu}. The EP step constructs a cavity distribution by removing one factor, forms a tilted distribution by multiplying the cavity with the corresponding exact message, and then projects this tilted distribution back onto the exponential family by matching moments (i.e., minimizing KL divergence). This yields updated natural parameters η_{+ wu} (or η_{+ ∘ u} for the node potential). Because the proposal belongs to a tractable family (the authors use Gaussians in experiments), sampling N particles from q_u is cheap and exact.

Given the particles {x_i^u}{i=1}^N, importance weights are computed as w_i^{uv}=cM{uv}(x_i^u)/q_u(x_i^u), where cM_{uv}(x_i^u)=B_u(x_i^u)/b_m^{vu}(x_i^u) and B_u(x_i^u)=ψ_u(x_i^u)∏{w∈Γ_u}b_m^{wu}(x_i^u). The outgoing message to neighbour v is then approximated by a weighted mixture b_m^{uv}(x_v)=∑{i=1}^N w_i^{uv} ψ_{uv}(x_i^u, x_v). This construction mirrors the PBP message form but, crucially, the sampling is exact and the estimator is provably consistent as N→∞.

The naive implementation requires O(|Γ_u| N²) operations per node because evaluating each weight involves summing over N mixture components of each incoming message. To reduce this cost, the authors adopt a technique from related work: they draw M (with M≪N) indices from a multinomial distribution defined by the normalized weights and evaluate only the corresponding M mixture components. This reduces the per‑node cost to O(|Γ_u| M N). By choosing M=O(log N), the overall algorithm achieves sub‑quadratic complexity while preserving consistency.

Experimental evaluation focuses on two small graph topologies—a 3×3 grid and a tree—where node and edge potentials are constructed from mixtures of Gaussian, Gumbel, and Laplace distributions, yielding multimodal, skewed, and non‑integrable beliefs. The ground truth is obtained via deterministic LBP on a fine grid. EPBP (with Gaussian proposals) is compared against the original PBP implementation (20‑step Metropolis‑Hastings MCMC for each proposal). Results show that EPBP attains significantly lower mean L₁ error across all nodes and converges as the number of particles grows, whereas PBP’s error plateaus due to biased sampling. Moreover, EPBP runs orders of magnitude faster; even the sub‑quadratic variant retains most of the accuracy while further reducing runtime. A simple image denoising task demonstrates that EPBP scales to practical applications.

In summary, the paper makes three key contributions: (1) an EP‑driven adaptive proposal mechanism that brings the proposal distribution close to the current belief without requiring costly MCMC; (2) a particle‑based LBP estimator that is unbiased and consistent, overcoming the limitations of existing non‑parametric BP methods; and (3) a sub‑quadratic implementation that makes the approach feasible for larger continuous‑state MRFs. The method is broadly applicable to domains such as tracking, sensor networks, and image restoration where continuous variables and complex potentials are common.