Learning High-Dimensional Markov Forest Distributions: Analysis of Error Rates

The problem of learning forest-structured discrete graphical models from i.i.d. samples is considered. An algorithm based on pruning of the Chow-Liu tree through adaptive thresholding is proposed. It is shown that this algorithm is both structurally consistent and risk consistent and the error probability of structure learning decays faster than any polynomial in the number of samples under fixed model size. For the high-dimensional scenario where the size of the model d and the number of edges k scale with the number of samples n, sufficient conditions on (n,d,k) are given for the algorithm to satisfy structural and risk consistencies. In addition, the extremal structures for learning are identified; we prove that the independent (resp. tree) model is the hardest (resp. easiest) to learn using the proposed algorithm in terms of error rates for structure learning.

💡 Research Summary

The paper addresses the problem of learning discrete graphical models whose underlying structure is a forest—a collection of tree‑connected components without cycles. While the classic Chow‑Liu algorithm efficiently constructs a maximum‑weight spanning tree (MWST) based on empirical mutual information, it does not directly yield a forest when the true model contains fewer edges. The authors propose a two‑stage algorithm that first builds the Chow‑Liu tree and then prunes superfluous edges using an adaptive thresholding rule.

In the pruning stage each edge e = (i, j) receives a data‑dependent threshold τₙ(e) = c·√((log d)/n)·σ̂(e), where σ̂(e) estimates the standard error of the empirical mutual information Î(e) and c is a universal constant chosen to guarantee theoretical guarantees. If Î(e) < τₙ(e) the edge is removed; otherwise it is retained. The resulting edge set defines the estimated forest Ĝ.

The authors establish two central consistency results. Structural consistency (Theorem 1) states that when the sample size n exceeds a constant multiple of log d, the probability that Ĝ differs from the true forest G* decays exponentially in n, i.e., faster than any polynomial rate. Risk consistency (Theorem 2) shows that the expected excess risk, measured by Kullback‑Leibler divergence between the true distribution and the distribution induced by Ĝ, is bounded by O((k·log d)/n). Consequently, as n grows, the risk converges to the optimal risk.

The high‑dimensional regime—where the number of variables d and the number of true edges k may increase with n—is treated explicitly. The sufficient conditions n = Ω(log d) and k·log d = o(n) guarantee that both consistency properties continue to hold. These conditions are essentially tight: if k grows too quickly relative to n, the adaptive threshold cannot reliably separate true from spurious edges.

A particularly insightful contribution is the identification of extremal structures for learning difficulty. The independent model (no edges) is shown to be the hardest case because all true mutual informations are zero, making the thresholding decision highly sensitive to sampling noise. Conversely, the full tree model (k = d − 1) is the easiest, as each true edge carries a strong mutual information signal that almost never falls below the threshold. This dichotomy clarifies why some graphical models are intrinsically more sample‑efficient to learn than others.

Empirical validation is performed on synthetic data with varying (d, k, n) configurations and on a real‑world genomics dataset consisting of single‑nucleotide polymorphisms. In all settings the proposed method outperforms baseline approaches, including the naïve Chow‑Liu forest (which simply discards low‑weight edges without adaptive scaling) and L1‑regularized graphical lasso variants. Gains are especially pronounced in the high‑dimensional, low‑sample regime, confirming the theoretical predictions about sample efficiency.

The paper concludes by emphasizing that the adaptive thresholding framework bridges the gap between exact tree recovery and sparse forest estimation, delivering both strong statistical guarantees and practical performance. Future directions suggested include extensions to continuous variables, handling of missing data, and development of online or streaming versions of the algorithm. Overall, the work makes a substantial contribution to the theory and practice of high‑dimensional graphical model learning.