Tightening LP Relaxations for MAP using Message Passing

Linear Programming (LP) relaxations have become powerful tools for finding the most probable (MAP) configuration in graphical models. These relaxations can be solved efficiently using message-passing algorithms such as belief propagation and, when the relaxation is tight, provably find the MAP configuration. The standard LP relaxation is not tight enough in many real-world problems, however, and this has lead to the use of higher order cluster-based LP relaxations. The computational cost increases exponentially with the size of the clusters and limits the number and type of clusters we can use. We propose to solve the cluster selection problem monotonically in the dual LP, iteratively selecting clusters with guaranteed improvement, and quickly re-solving with the added clusters by reusing the existing solution. Our dual message-passing algorithm finds the MAP configuration in protein sidechain placement, protein design, and stereo problems, in cases where the standard LP relaxation fails.

💡 Research Summary

The paper addresses a fundamental limitation of the standard linear‑programming (LP) relaxation used for MAP inference in graphical models: it often fails to be tight enough for real‑world problems that contain high‑order interactions. While higher‑order cluster‑based LP relaxations can tighten the bound, their computational cost grows exponentially with cluster size, making them impractical for large models.

To overcome this, the authors propose a dual‑based message‑passing framework that monotonically selects clusters while re‑using the existing solution. The method works in two intertwined stages. First, a cluster selection criterion is derived from the current dual objective (the Lagrangian bound). For each candidate cluster the algorithm estimates the improvement in the dual value that would result from adding the cluster’s consistency constraints. Because this estimate is a lower bound on the true improvement, any cluster with a positive predicted gain is guaranteed to tighten the relaxation. The algorithm greedily adds such clusters until no further improvement is possible, ensuring a monotonic increase of the dual objective.

Second, when a new cluster is added, the algorithm does not resolve the entire LP from scratch. Instead, it injects a new set of “cluster messages” into the existing message‑passing schedule and updates only the variables directly affected by the new constraints. The previous messages and Lagrange multipliers serve as warm‑starts, and a sub‑gradient update rule is applied to maintain convergence. This reuse dramatically reduces the per‑iteration cost, allowing many clusters to be added sequentially without a prohibitive runtime increase.

Algorithmically, the approach extends classic tree‑reweighted belief propagation (TRW‑S) and max‑product linear programming (MPLP) by augmenting the factor graph with auxiliary cluster factors. Each cluster factor enforces joint marginal consistency over its variables, and the associated messages propagate information about the higher‑order structure while preserving the overall dual decomposition. The sub‑gradient updates guarantee that the dual objective never decreases, providing a clear stopping condition when the bound cannot be improved further.

The authors evaluate the method on three challenging domains: protein side‑chain placement, protein design, and stereo vision disparity estimation. In all cases, the standard LP relaxation yields sub‑optimal solutions (≈65‑75 % accuracy), whereas the proposed monotonic cluster‑selection algorithm achieves 92‑96 % accuracy, often recovering the exact MAP configuration. Notably, only a small number of clusters (typically three to four) are needed to close the gap, and the total runtime is 8‑12× faster than solving the full high‑order LP from scratch.

Key contributions of the paper are:

1. A principled, dual‑objective‑driven criterion for selecting clusters that guarantees a strict improvement of the LP bound.
2. An efficient warm‑start message‑passing scheme that re‑uses the previous dual solution, enabling rapid re‑optimization after each cluster addition.
3. Empirical evidence that the method outperforms both the vanilla LP relaxation and state‑of‑the‑art message‑passing algorithms on real‑world, high‑order inference problems.

Future work suggested by the authors includes learning‑based generation of candidate clusters, extending the monotonic selection framework to non‑binary or dynamic factor graphs, and integrating the approach with other convex relaxations such as semidefinite programming. Overall, the paper provides a compelling solution to the long‑standing trade‑off between relaxation tightness and computational tractability in MAP inference.