On the trade-off between complexity and correlation decay in structural learning algorithms

We consider the problem of learning the structure of Ising models (pairwise binary Markov random fields) from i.i.d. samples. While several methods have been proposed to accomplish this task, their relative merits and limitations remain somewhat obscure. By analyzing a number of concrete examples, we show that low-complexity algorithms often fail when the Markov random field develops long-range correlations. More precisely, this phenomenon appears to be related to the Ising model phase transition (although it does not coincide with it).

💡 Research Summary

The paper investigates the fundamental trade‑off between computational (algorithmic) complexity and sample complexity in learning the structure of Ising models—binary pairwise Markov random fields—from independent and identically distributed (i.i.d.) samples. While the Ising model is a canonical example of an exponential family and is theoretically identifiable for any positive interaction strength θ, practical learning must respect realistic constraints on the number of samples and on computational resources.

The authors focus on three representative low‑complexity learning procedures: (i) a simple thresholding algorithm that declares an edge whenever the empirical pairwise correlation exceeds a chosen threshold τ; (ii) a conditional independence test (the Bresler‑Mossel‑Sly method) that checks, for each node, whether another node is independent of it given its Markov blanket; and (iii) regularized logistic regression (RLR) as introduced by Ravikumar, Wainwright, and Lafferty, which solves an ℓ₁‑penalized logistic regression for each node to recover its neighborhood.

A central concept is the uniqueness threshold θ₍uniq₎(Δ) = atanh(1/(Δ − 1)), where Δ denotes the maximum degree of the underlying graph. For interaction strengths below this threshold, Gibbs sampling mixes rapidly, and variables that are far apart in the graph are essentially independent (strong correlation decay). Above the threshold, long‑range correlations persist, making the inference problem substantially harder.

Main theoretical contributions

1. Thresholding – For trees, setting τ = (tanh θ + tanh² θ)/2 yields a sample complexity n = O(log p) (p = number of vertices) sufficient for exact recovery. For general graphs with bounded degree Δ, if θ < atanh(1/(2Δ)), a threshold τ = (tanh θ + 1/(2Δ))/2 achieves the same logarithmic sample requirement. Thus, when correlation decays fast enough, a naïve O(p²n) algorithm is statistically optimal.

2. Conditional independence test – When θ < θ₍uniq₎(Δ), the test succeeds with sample complexity O(Δ log p) and computational cost O(pΔn). If θ exceeds the uniqueness threshold, the test’s statistical power deteriorates dramatically, leading to a sharp increase in required samples.

3. Regularized logistic regression – Prior work guarantees O(log p) sample complexity for bounded‑degree graphs provided θ < θ₍uniq₎(Δ). The present analysis shows that for graphs where θ is above the uniqueness threshold, RLR can systematically mis‑identify the structure. A concrete counterexample uses two families of graphs, Gₚ and G′ₚ: Gₚ contains many weak indirect paths between nodes 1 and 2, while G′ₚ has a single strong direct edge. By scaling θ = O(1/√p), all low‑dimensional marginals of the two models differ only by O(1/√p), making them indistinguishable to any algorithm that relies solely on a small number of low‑order statistics. In this regime, RLR reconstructs G′ₚ (the wrong graph) whereas a global statistic (e.g., the full empirical covariance matrix) would succeed with a sample size that grows linearly with p.

Implications and broader picture
The paper demonstrates that low‑complexity algorithms succeed only in the “correlation‑decay regime” (θ < const / Δ). When the interaction strength is large relative to the degree, long‑range correlations can masquerade as direct edges, causing algorithms that examine only local or low‑dimensional statistics to fail unless the number of samples scales with the graph size. This establishes a precise quantitative relationship between the physical parameters of the Ising model (θ, Δ) and the algorithmic resources required for exact structure recovery.

The authors also discuss connections to phase‑transition phenomena: the uniqueness threshold is related but not identical to the critical temperature θ₍crit₎ at which the model undergoes a thermodynamic phase transition. Empirically, the algorithms examined tend to break down for θ ≫ const / Δ, a regime that often coincides with the onset of slow mixing for Gibbs samplers.

Finally, the paper raises open questions, notably whether there exist polynomial‑time algorithms that can provably recover graph structure in the strongly dependent regime (θ ≫ θ₍uniq₎). While some heuristic methods (e.g., adaptive clustering) appear promising in practice, a rigorous analysis remains lacking. The work thus provides both a solid theoretical foundation for understanding existing methods and a roadmap for future research on high‑complexity, low‑sample‑size structure learning in strongly correlated graphical models.