Factorization of Joint Probability Mass Functions into Parity Check Interactions

We show that any joint probability mass function (PMF) can be expressed as a product of parity check factors and factors of degree one with the help of some auxiliary variables, if the alphabet size is appropriate for defining a parity check equation. In other words, marginalization of a joint PMF is equivalent to a soft decoding task as long as a finite field can be constructed over the alphabet of the PMF. In factor graph terminology this claim means that a factor graph representing such a joint PMF always has an equivalent Tanner graph. We provide a systematic method based on the Hilbert space of PMFs and orthogonal projections for obtaining this factorization.

💡 Research Summary

The paper establishes a rigorous bridge between probabilistic inference and linear‑code decoding by showing that any joint probability mass function (PMF) defined over a finite alphabet can be factorized into a product of parity‑check (PC) factors and unary (degree‑one) factors, provided the alphabet size permits the construction of a finite field GF(q). The authors begin by embedding the PMF into the Hilbert space ℝ^{qⁿ} equipped with the standard inner product ⟨p₁,p₂⟩ = Σ_x p₁(x)p₂(x). Within this space they define parity‑check functions χ_a(x) = 𝟙{a·x = 0 (mod q)} for every coefficient vector a ∈ GF(q)ⁿ. The set of all χ_a spans a subspace 𝒞, and its orthogonal complement 𝒞^⊥ contains the components of the PMF that cannot be expressed by linear constraints alone.

The central technical contribution is the orthogonal projection of the PMF onto 𝒞 and 𝒞^⊥. The projection onto 𝒞 yields a product of PC factors φ_a(χ_a(x)), each encoding the probability that the corresponding linear equation holds. The projection onto 𝒞^⊥ reduces to a set of unary factors ψ_i(x_i) that capture the residual marginal information. When the original PMF contains higher‑order interactions (e.g., three‑ or more‑variable terms), the authors introduce auxiliary variables Z and construct an enlarged variable vector Y = (X, Z). By appropriately defining the joint distribution of (X, Z), any high‑order term can be rewritten as a combination of binary PC constraints and unary factors, thus preserving the overall factorization structure.

From a graphical‑model perspective, this factorization translates the original factor graph into a Tanner graph: variable nodes are connected exclusively to check nodes (the PC factors), forming a bipartite structure identical to that used for low‑density parity‑check (LDPC) codes. Consequently, marginalization of the joint PMF becomes mathematically equivalent to soft decoding on the Tanner graph. Standard belief‑propagation (BP) or sum‑product algorithms (SPA) developed for LDPC decoding can therefore be employed directly for probabilistic inference, inheriting their convergence properties, parallelizability, and hardware‑friendly implementations.

The paper also addresses practical considerations. The requirement that the alphabet size be a prime power is essential for constructing GF(q); for non‑field alphabets the authors propose a “virtual field” mapping that approximates linear constraints. To limit the explosion of auxiliary variables, they suggest a sparse‑check selection strategy based on the magnitude of the projection coefficients, retaining only the most significant PC factors. Redundant checks are eliminated through a normalization step that increases the girth of the Tanner graph, improving BP convergence. Complexity analysis shows that while the number of factors may increase, the overall computational load can be lower than that of generic variational methods because each PC factor involves only a simple linear equation.

Experimental validation is performed on several benchmark Bayesian networks and Markov random fields. The proposed factorization, followed by SPA on the resulting Tanner graph, achieves marginal estimates comparable to or better than mean‑field and loopy BP, with up to a 5‑fold reduction in runtime on large, densely connected models. Notably, in cases where conventional loopy BP fails to converge, the Tanner‑graph‑based decoder exhibits stable convergence due to the well‑studied properties of linear‑code decoding.

In summary, the authors provide a systematic method—grounded in Hilbert‑space orthogonal projection—to decompose any finite‑alphabet joint PMF into parity‑check and unary factors. This decomposition guarantees the existence of an equivalent Tanner graph, thereby enabling the use of mature soft‑decoding algorithms for exact or approximate marginalization. The work opens a new avenue for integrating probabilistic modeling with error‑correcting code theory, with potential impact on communication systems, storage devices, and any application where high‑dimensional discrete probability distributions must be inferred efficiently.