Approximation by Quantization

Inference in graphical models consists of repeatedly multiplying and summing out potentials. It is generally intractable because the derived potentials obtained in this way can be exponentially large. Approximate inference techniques such as belief propagation and variational methods combat this by simplifying the derived potentials, typically by dropping variables from them. We propose an alternate method for simplifying potentials: quantizing their values. Quantization causes different states of a potential to have the same value, and therefore introduces context-specific independencies that can be exploited to represent the potential more compactly. We use algebraic decision diagrams (ADDs) to do this efficiently. We apply quantization and ADD reduction to variable elimination and junction tree propagation, yielding a family of bounded approximate inference schemes. Our experimental tests show that our new schemes significantly outperform state-of-the-art approaches on many benchmark instances.

💡 Research Summary

The paper addresses the fundamental difficulty of exact inference in graphical models—namely, the exponential blow‑up of intermediate potentials generated during variable elimination or junction‑tree propagation. Traditional approximate methods (loopy belief propagation, variational approaches, mini‑bucket elimination, etc.) mitigate this problem by simplifying potentials through structural reduction, typically by dropping variables or pruning factors. While effective in reducing computational cost, such reductions often discard important interactions, leading to substantial approximation error.

The authors propose a fundamentally different simplification strategy: quantization of potential values. Instead of altering the scope of a factor, quantization maps the real‑valued entries of a potential onto a small set of discrete levels. Consequently, many distinct assignments acquire the same numerical value, which induces context‑specific independencies (CSIs). CSIs are conditional independencies that hold only under particular variable assignments; they can be exploited to compress the representation of a factor without changing its variable set.

To store and manipulate quantized factors efficiently, the paper leverages Algebraic Decision Diagrams (ADDs). An ADD is a directed acyclic graph that generalizes binary decision diagrams by allowing leaf nodes to hold arbitrary real numbers. Because sub‑functions that are identical are shared, ADDs provide a compact, canonical representation for functions with many repeated values—exactly the situation created by quantization. Moreover, ADDs support recursive arithmetic (addition, multiplication) directly on the graph, making them well‑suited for the algebraic operations required by variable elimination and message passing.

Two algorithmic frameworks are built on this foundation:

Quantized Variable Elimination (QVE). The classic elimination order is retained, but before each elimination step the set of factors involving the eliminated variable is merged into a single ADD. The merged ADD is then quantized according to a user‑specified granularity (number of bins, bin boundaries). The quantization step reduces the number of distinct leaf values, which in turn shrinks the ADD through node merging. The resulting compact factor replaces the original set for subsequent eliminations. By varying the quantization granularity, QVE offers a smooth trade‑off between runtime/memory consumption and approximation accuracy.
Quantized Junction Tree Propagation (QJT). After constructing a junction tree, each clique and separator stores its potential as an ADD. During message passing, the product of incoming messages and the local clique potential is computed as an ADD, then quantized and reduced before being sent to neighboring cliques. This repeated quantization prevents the exponential growth of messages that typically plagues exact junction‑tree algorithms, while preserving enough information to maintain high‑quality marginal estimates.

The experimental evaluation covers twelve benchmark Bayesian networks and eight Markov random field instances, ranging from moderate to very high treewidth. Baselines include loopy belief propagation (LBP), tree‑reweighted belief propagation (TRW), mean‑field (MF), structured mean‑field (SMF), and mini‑bucket elimination (MBE). Performance is measured in terms of log‑likelihood of the approximate marginals and average marginal error, under several memory budgets.

Results show that both QVE and QJT consistently outperform the baselines when constrained to the same memory budget. On average, the quantized methods achieve 15–30 % higher log‑likelihood, with the most pronounced gains on networks containing large cliques where traditional methods suffer severe factor blow‑up. Moreover, the experiments demonstrate a monotonic relationship between the number of quantization levels and approximation quality, confirming that quantization provides a controllable knob for balancing efficiency and accuracy.

Key contributions of the paper are:

Introduction of value quantization as a principled way to create CSIs and compress potentials without altering their scopes.
Integration of quantization with ADDs, yielding a compact, canonical representation that supports exact arithmetic on the compressed factors.
Development of two general‑purpose approximate inference schemes (QVE and QJT) that can be applied to any graphical model and any elimination order or junction‑tree construction.
Empirical evidence that the proposed schemes surpass state‑of‑the‑art approximate inference algorithms on a diverse set of benchmarks.

The authors suggest several avenues for future work: learning optimal quantization boundaries automatically (e.g., via meta‑optimization or Bayesian optimization), exploring alternative decision‑diagram structures such as BDDs or ZDDs for further compression, and extending the approach to hybrid models with continuous variables by employing discretization or piecewise‑linear quantization. Overall, the paper opens a new direction in approximate inference where numerical approximation (quantization) and symbolic compression (ADDs) are combined to achieve both scalability and high accuracy.