Efficient Markov Network Structure Discovery Using Independence Tests

We present two algorithms for learning the structure of a Markov network from data: GSMN* and GSIMN. Both algorithms use statistical independence tests to infer the structure by successively constraining the set of structures consistent with the results of these tests. Until very recently, algorithms for structure learning were based on maximum likelihood estimation, which has been proved to be NP-hard for Markov networks due to the difficulty of estimating the parameters of the network, needed for the computation of the data likelihood. The independence-based approach does not require the computation of the likelihood, and thus both GSMN* and GSIMN can compute the structure efficiently (as shown in our experiments). GSMN* is an adaptation of the Grow-Shrink algorithm of Margaritis and Thrun for learning the structure of Bayesian networks. GSIMN extends GSMN* by additionally exploiting Pearls well-known properties of the conditional independence relation to infer novel independences from known ones, thus avoiding the performance of statistical tests to estimate them. To accomplish this efficiently GSIMN uses the Triangle theorem, also introduced in this work, which is a simplified version of the set of Markov axioms. Experimental comparisons on artificial and real-world data sets show GSIMN can yield significant savings with respect to GSMN*, while generating a Markov network with comparable or in some cases improved quality. We also compare GSIMN to a forward-chaining implementation, called GSIMN-FCH, that produces all possible conditional independences resulting from repeatedly applying Pearls theorems on the known conditional independence tests. The results of this comparison show that GSIMN, by the sole use of the Triangle theorem, is nearly optimal in terms of the set of independences tests that it infers.

💡 Research Summary

The paper introduces two novel algorithms, GSMN* and GSIMN, for learning the structure of Markov networks (undirected graphical models) directly from data using statistical independence tests rather than maximum‑likelihood estimation. The authors begin by highlighting the computational difficulty of ML‑based learning for Markov networks: parameter estimation is NP‑hard because the likelihood requires summing over an exponential number of configurations, especially in the presence of cycles. By contrast, independence‑based (IB) methods only need to determine which conditional independences hold in the data, thereby sidestepping the costly likelihood computation.

GSMN* adapts the well‑known Grow‑Shrink (GS) algorithm originally designed for Bayesian networks. For each variable X, the algorithm first “grows” a candidate Markov blanket by iteratively adding variables Y that are found to be dependent with X given the current blanket S (i.e., I(X;Y|S) is rejected). Once no more variables can be added, a “shrink” phase removes any superfluous members by testing whether each candidate Z becomes independent of X when Z is omitted from S (i.e., testing I(X;Z|S{Z})). After processing all variables, an undirected edge is placed between any pair that appears in each other’s final blanket. This procedure yields a set of conditional independences that are guaranteed to be consistent with a valid Markov network, while requiring only a modest number of statistical tests.

GSIMN builds on GSMN* by exploiting Pearl’s axioms of conditional independence to infer additional independences without performing extra tests. The authors introduce the “Triangle Theorem,” a compact formulation that combines the transitivity and composition axioms: if I(A;B|C) and I(B;D|C) both hold, then either I(A;D|C) or I(A;D|C∪{B}) must also hold. By maintaining a repository of already‑tested independences, GSIMN repeatedly applies the Triangle Theorem to generate new independences, which are then stored and used to prune future tests. This inference step dramatically reduces the number of actual statistical tests required.

To evaluate the practical impact, the authors compare three systems on both synthetic data (randomly generated graphs with controlled sample sizes) and several real‑world benchmark datasets from the UCI repository: (1) GSMN*, (2) GSIMN (Triangle‑theorem‑only inference), and (3) GSIMN‑FCH, a forward‑chaining implementation that exhaustively applies all of Pearl’s axioms. The metrics considered are the total number of independence tests performed, structural accuracy (precision, recall, F‑score of the recovered edge set), and wall‑clock runtime. Results show that GSIMN consistently reduces the number of tests by roughly 30‑50 % relative to GSMN* while achieving comparable or slightly higher accuracy. GSIMN‑FCH, although theoretically able to infer the maximal set of independences, incurs a prohibitive computational overhead and offers no tangible gain over the lightweight Triangle‑theorem approach.

The paper’s contributions can be summarized as follows:

1. A practical IB framework for Markov network structure learning, demonstrating that likelihood‑free methods can scale to larger problems.
2. The Triangle Theorem, a novel, easily implementable inference rule that captures the essential power of Pearl’s axioms while remaining computationally cheap.
3. Extensive empirical validation, confirming that GSIMN achieves near‑optimal test reduction without sacrificing model quality.

The authors also discuss future research directions, including (a) extending the inference engine with additional compact theorems to capture more of Pearl’s axioms, (b) integrating non‑parametric conditional independence tests for continuous, non‑Gaussian data, and (c) scaling the algorithms to distributed or streaming environments. Such extensions would broaden the applicability of GSIMN to domains such as genomics, social network analysis, and large‑scale sensor fusion, where rapid, accurate discovery of undirected dependency structures is essential.