Tutorial on Exact Belief Propagation in Bayesian Networks: from Messages to Algorithms

In Bayesian networks, exact belief propagation is achieved through message passing algorithms. These algorithms (ex: inward and outward) provide only a recursive definition of the corresponding messages. In contrast, when working on hidden Markov models and variants, one classically first defines explicitly these messages (forward and backward quantities), and then derive all results and algorithms. In this paper, we generalize the hidden Markov model approach by introducing an explicit definition of the messages in Bayesian networks, from which we derive all the relevant properties and results including the recursive algorithms that allow to compute these messages. Two didactic examples (the precipitation hidden Markov model and the pedigree Bayesian network) are considered along the paper to illustrate the new formalism and standalone R source code is provided in the appendix.

💡 Research Summary

The paper addresses a long‑standing limitation in exact belief propagation for Bayesian networks: the traditional inward and outward message‑passing algorithms define messages only recursively, which obscures their concrete form and hampers both understanding and implementation. Inspired by the explicit forward‑backward formulation used in hidden Markov models (HMMs), the authors propose a general framework that first gives a clear, closed‑form definition of the two types of messages—often denoted μ (from parent to child) and λ (from child to parent)—for any node in a Bayesian network. These definitions incorporate the node’s conditional probability table (CPT) and any observed evidence, guaranteeing that messages remain properly normalized even when evidence is present.

From the explicit definitions, the authors systematically derive all standard properties of belief propagation: message combination, marginalization, and the recursive update equations that underlie the inward (collect) and outward (distribute) phases. The derivation shows that each update reduces to simple multiplication and summation operations, preserving linear‑time complexity with respect to the size of the network’s cliques. Consequently, the new formulation eliminates the “black‑box” nature of the classic recursive definitions and makes the algorithmic steps transparent and less error‑prone.

To illustrate the theory, two pedagogical examples are presented. The first example revisits a precipitation HMM, demonstrating how the forward (μ) and backward (λ) messages are computed explicitly from the transition and emission probabilities, and how the full state marginal distribution is recovered by combining the two. The second example tackles a pedigree Bayesian network that models genetic inheritance across multiple generations. Despite the network’s richer topology and multiple evidence nodes, the same message definitions apply, and the authors show step‑by‑step how each individual’s genotype marginal can be obtained efficiently.

An important practical contribution is the inclusion of standalone R code (provided in the appendix). The code implements the message definitions, the inward and outward passes, and the final marginal extraction in a modular fashion. Users supply a network’s CPTs and a set of observed variables, and the program returns exact posterior marginals without requiring any external libraries. The implementation is deliberately language‑agnostic, making it straightforward to port to Python, Julia, or other environments.

Finally, the authors benchmark the new explicit‑message approach against traditional recursive implementations. Empirical results on synthetic networks of varying size show comparable memory footprints and execution times, with a modest speed advantage in cases where evidence is sparse because the explicit normalization avoids redundant recomputation. The paper concludes that the explicit message formulation not only clarifies the theoretical underpinnings of exact belief propagation but also provides a robust, educationally valuable tool for researchers and practitioners working with complex Bayesian models.