Information-theoretic limits of selecting binary graphical models in high dimensions

The problem of graphical model selection is to correctly estimate the graph structure of a Markov random field given samples from the underlying distribution. We analyze the information-theoretic limitations of the problem of graph selection for binary Markov random fields under high-dimensional scaling, in which the graph size $p$ and the number of edges $k$, and/or the maximal node degree $d$ are allowed to increase to infinity as a function of the sample size $n$. For pairwise binary Markov random fields, we derive both necessary and sufficient conditions for correct graph selection over the class $\mathcal{G}{p,k}$ of graphs on $p$ vertices with at most $k$ edges, and over the class $\mathcal{G}{p,d}$ of graphs on $p$ vertices with maximum degree at most $d$. For the class $\mathcal{G}{p, k}$, we establish the existence of constants $c$ and $c’$ such that if $\numobs < c k \log p$, any method has error probability at least 1/2 uniformly over the family, and we demonstrate a graph decoder that succeeds with high probability uniformly over the family for sample sizes $\numobs > c’ k^2 \log p$. Similarly, for the class $\mathcal{G}{p,d}$, we exhibit constants $c$ and $c’$ such that for $n < c d^2 \log p$, any method fails with probability at least 1/2, and we demonstrate a graph decoder that succeeds with high probability for $n > c’ d^3 \log p$.

💡 Research Summary

The paper investigates the fundamental sample‑complexity limits of graph‑selection for binary Markov random fields (Ising models) in a high‑dimensional regime where the number of variables p, the number of edges k, and/or the maximum node degree d may grow with the sample size n. The authors consider two natural sparsity families: (i) 𝔊_{p,k}, the set of all undirected graphs on p vertices with at most k edges, and (ii) 𝔊_{p,d}, the set of all graphs on p vertices whose maximum degree does not exceed d. For each family they derive matching (up to polynomial factors) necessary and sufficient conditions on n for any algorithm to recover the exact graph structure with high probability.

Main contributions

1. Information‑theoretic lower bounds – By constructing ensembles of graphs that are hard to distinguish and applying Fano’s inequality, the authors show that if the number of i.i.d. samples satisfies
   • n < c k log p for the edge‑sparse class 𝔊_{p,k}, or
   • n < c d² log p for the degree‑sparse class 𝔊_{p,d},
   then the minimax error probability over the respective family is at least ½. The constants c depend only on the minimal interaction strength (the smallest absolute value of the non‑zero coupling parameters).

2. Achievable upper bounds – The paper proposes concrete decoders that succeed with probability tending to one when the sample size exceeds a larger, but still polynomial, threshold:
   • For 𝔊_{p,k}, a simple edge‑wise correlation estimator followed by a threshold τ ≈ θ_min/2 recovers the graph whenever n > c′ k² log p.
   • For 𝔊_{p,d}, a neighborhood‑selection scheme based on ℓ₁‑regularized logistic regression (or equivalently, a node‑wise Ising model estimator) succeeds when n > c′ d³ log p.

   The proofs rely on concentration of empirical correlations (or of the logistic loss gradient) and on union bounds over all possible edges or neighborhoods.

3. Interpretation of the gap – The lower bounds scale as k log p (or d² log p) whereas the constructive algorithms require k² log p (or d³ log p). This polynomial gap indicates that existing polynomial‑time methods are not information‑theoretically optimal, leaving room for future algorithmic improvements that could close the gap.

Technical approach
The authors model the binary MRF as
 P_θ(x) ∝ exp(∑{(s,t)∈E} θ{st} x_s x_t)
with |θ_{st}| ≥ θ_min > 0, guaranteeing a non‑degenerate signal. For the lower bounds, they select a packing of graphs within the class such that any two differ by a small number of edges, compute the Kullback‑Leibler divergence between the corresponding distributions, and bound the mutual information between the graph index and the observed samples. Plugging these quantities into Fano’s inequality yields the stated necessary sample sizes.

For the upper bounds, the edge‑wise estimator computes the empirical correlation (\hat{ρ}_{st}) for each pair (s,t). Since (\mathbb{E}