Parallelizable Feynman-Kac Models for Universal Probabilistic Programming

We study provably correct and efficient instantiations of Sequential Monte Carlo (SMC) inference in the context of formal operational semantics of Probabilistic Programs (PPs). We focus on universal PPs featuring sampling from arbitrary measures and conditioning/reweighting in unbounded loops. We first equip Probabilistic Program Graphs (PPGs), an automata-theoretic description format of PPs, with an expectation-based semantics over infinite execution traces, which also incorporates trace weights. We then prove a finite approximation theorem that provides bounds to this semantics based on expectations taken over finite, fixed-length traces. This enables us to frame our semantics within a Feynman-Kac (FK) model, and ensures the consistency of the Particle Filtering (PF) algorithm, an instance of SMC, with respect to our semantics. Building on these results, we introduce VPF, a vectorized version of the PF algorithm tailored to PPGs and our semantics. Experiments conducted with a proof-of-concept implementation of VPF show very promising results compared to state-of-the-art PP inference tools.

💡 Research Summary

The paper tackles a fundamental gap between the formal semantics of universal probabilistic programming languages (PPLs) and scalable inference algorithms. It introduces Probabilistic Program Graphs (PPGs), an automata‑theoretic representation of probabilistic programs that can express sampling from arbitrary continuous, discrete, or mixed measures, as well as conditioning inside unbounded loops. Each transition in a PPG is modeled by a parametric Markov kernel (the sampling distribution) and a score function (the re‑weighting factor).

On the semantic side, the authors define an expectation‑based semantics over infinite execution traces. A trace is an infinite sequence of transitions; its weight is the product of all score function values encountered along the trace. This infinite‑trace semantics captures the exact posterior distribution of program outputs in a measure‑theoretic sense. Because directly handling infinite traces is infeasible, the paper proves a finite‑approximation theorem. The theorem states that for any prefix‑closed functional (i.e., a function whose value depends only on a finite prefix of the trace), the expectation under the infinite‑trace measure can be approximated arbitrarily well by expectations over finite‑length traces. The error bound shrinks as the truncation length grows.

Using this theorem, the authors embed the PPG semantics into a finite‑horizon Feynman‑Kac (FK) model, a well‑studied formalism for state‑space models with observations. In the FK setting, Sequential Monte Carlo (SMC) methods—specifically Particle Filtering (PF)—are known to be consistent: as the number of particles N → ∞, the empirical particle distribution converges to the true FK measure. Consequently, the consistency of PF for PPGs follows immediately from the FK embedding.

The algorithmic contribution is a vectorized version of PF, called VPF. VPF exploits SIMD parallelism by applying the transition kernel and the score function simultaneously to the entire particle vector at each step. Resampling is performed after every transition, guaranteeing that all particles remain aligned, which eliminates the need for complex bookkeeping required in functional‑style SMC implementations. The authors implement VPF in TensorFlow, leveraging both CPU and GPU tensor operations.

Empirical evaluation compares VPF against state‑of‑the‑art PPL inference engines (e.g., Pyro, Stan, Gen) on a suite of challenging models that feature unbounded loops, mixed continuous/discrete distributions, and heavy conditioning. Results show that VPF achieves speed‑ups of 3–5× on typical workloads and scales efficiently to hundreds of thousands of particles, especially when GPU acceleration is used. Accuracy metrics such as Effective Sample Size (ESS) and mean‑squared error are comparable to or better than the baselines, confirming that the vectorized approach does not sacrifice statistical quality.

In summary, the paper makes three major contributions: (1) a clean operational semantics for PPGs based on infinite‑trace expectations, (2) a finite‑approximation theorem linking this semantics to FK models and establishing PF consistency, and (3) a SIMD‑friendly vectorized PF algorithm (VPF) with demonstrated practical scalability. By bridging rigorous measure‑theoretic foundations with high‑performance parallel inference, the work advances both the theory and practice of universal probabilistic programming.