On Learning Discrete Graphical Models Using Greedy Methods

In this paper, we address the problem of learning the structure of a pairwise graphical model from samples in a high-dimensional setting. Our first main result studies the sparsistency, or consistency in sparsity pattern recovery, properties of a forward-backward greedy algorithm as applied to general statistical models. As a special case, we then apply this algorithm to learn the structure of a discrete graphical model via neighborhood estimation. As a corollary of our general result, we derive sufficient conditions on the number of samples n, the maximum node-degree d and the problem size p, as well as other conditions on the model parameters, so that the algorithm recovers all the edges with high probability. Our result guarantees graph selection for samples scaling as n = Omega(d^2 log(p)), in contrast to existing convex-optimization based algorithms that require a sample complexity of \Omega(d^3 log(p)). Further, the greedy algorithm only requires a restricted strong convexity condition which is typically milder than irrepresentability assumptions. We corroborate these results using numerical simulations at the end.

💡 Research Summary

The paper tackles the problem of learning the structure of high‑dimensional discrete graphical models, specifically pairwise Markov random fields (MRFs), using a forward‑backward greedy algorithm. The authors first present a general analysis of this greedy procedure for arbitrary loss functions, establishing conditions under which the algorithm is sparsistent—that is, it recovers the exact support of a sparse parameter vector with high probability. The key technical assumptions are restricted strong convexity (RSC) and restricted smoothness (RSS) of the empirical loss, together with a bound on the ℓ∞ norm of the gradient at the true parameter (the “noise level”). Under these assumptions, Theorem 1 shows that if the stopping threshold εS is chosen appropriately (proportional to the squared noise level) and the true non‑zero coefficients are sufficiently large, the algorithm’s output satisfies three properties: (a) a concrete ℓ2 error bound, (b) no false exclusions (all true variables are selected), and (c) no false inclusions (no spurious variables are selected). The proof hinges on lemmas that relate the forward and backward steps to the RSC/RSS constants and on a bound that limits the size of the selected support to a multiple of the true sparsity.

The authors then specialize this general result to the case of binary Ising models (variables taking values in {−1,+1}). For each node r, they consider the conditional log‑likelihood of Xr given the rest of the variables as the loss function L(Θr). Applying the greedy algorithm independently to each node yields estimated neighborhoods (\widehat N(r)). The global edge set is formed by an “OR” rule (or alternatively an “AND” rule) over these neighborhoods. Theorem 2 demonstrates that, provided the maximum node degree is d, a sample size of order n = C·d²·log p (with C depending on the RSC/RSS constants) suffices for exact graph recovery with high probability. This improves upon the best known ℓ1‑regularized convex‑optimization approaches, which require n = Ω(d³·log p). Moreover, the required RSC condition is milder than the irrepresentability condition typically needed for ℓ1 methods, making the assumptions more realistic in practice.

Empirical experiments on synthetic Ising graphs and on real‑world datasets confirm the theoretical predictions. The greedy method achieves comparable or higher edge‑recovery accuracy than ℓ1‑regularized neighborhood selection while using significantly fewer samples. Computationally, each forward step involves a simple line search over a single coordinate, and each backward step checks the contribution of currently selected coordinates; thus the overall complexity scales roughly linearly in the number of variables, far below the polynomial costs (O(p⁴) or O(p⁶)) of solving large convex programs.

In summary, the paper makes three major contributions: (1) a unified sparsistency analysis for forward‑backward greedy algorithms under RSC/RSS, applicable beyond linear regression; (2) a concrete application to discrete graphical model selection that yields a sample complexity of O(d² log p), improving on existing methods; and (3) experimental validation showing both statistical and computational advantages. The work opens avenues for extending greedy techniques to multi‑state discrete models, non‑convex losses, and online learning settings.