A Generalization of the Chow-Liu Algorithm and its Application to Statistical Learning

We extend the Chow-Liu algorithm for general random variables while the previous versions only considered finite cases. In particular, this paper applies the generalization to Suzuki’s learning algorithm that generates from data forests rather than trees based on the minimum description length by balancing the fitness of the data to the forest and the simplicity of the forest. As a result, we successfully obtain an algorithm when both of the Gaussian and finite random variables are present.

💡 Research Summary

The paper presents a comprehensive generalization of the Chow‑Liu algorithm that removes the restriction to purely discrete random variables and makes the method applicable to continuous (Gaussian) variables as well as mixed discrete‑continuous settings. The classic Chow‑Liu approach builds a maximum‑weight spanning tree (MWST) where each edge weight is the mutual information between two variables; this tree approximates the full joint distribution while minimizing total entropy. However, the original formulation assumes that all variables have finite alphabets, which precludes direct use with real‑valued data.

To overcome this limitation, the authors define a unified mutual‑information estimator that works for three cases: (i) two discrete variables, (ii) two Gaussian variables, and (iii) a discrete–continuous pair. For Gaussian pairs, the mutual information reduces to a closed‑form expression involving the correlation coefficient derived from the covariance matrix. For mixed pairs, they combine the conditional density of the continuous variable given the discrete state with the marginal probability of the discrete variable, yielding a hybrid entropy term. These estimators are unbiased under standard regularity conditions and can be computed efficiently from sample covariances and empirical frequency tables.

With these generalized edge weights, the MWST can still be obtained using Kruskal’s or Prim’s algorithm, preserving the original computational complexity of O(N² log N) for N variables. The second major contribution is the integration of this generalized MWST into Suzuki’s Minimum Description Length (MDL) based forest learning algorithm. Suzuki’s method selects a forest rather than a single tree by balancing the log‑likelihood of the data given the graph against a penalty proportional to the number of edges (model complexity). In the original formulation, the log‑likelihood term is straightforward for discrete data but becomes ambiguous when continuous variables are present.

The authors resolve this by computing the log‑likelihood for each edge using the appropriate model: Gaussian likelihood for continuous‑continuous edges, multinomial likelihood for discrete‑discrete edges, and a conditional Gaussian likelihood for mixed edges. The MDL penalty remains a simple linear function of the edge count, ensuring that the trade‑off between fit and simplicity is comparable across variable types. The resulting algorithm proceeds as follows:

1. Estimate the generalized mutual information for every variable pair.
2. Construct the MWST using these weights.
3. Iterate over the edges of the tree; for each edge, evaluate the MDL score with and without the edge.
4. Remove the edge if its removal reduces the total MDL.
5. Repeat until no further MDL improvement is possible, yielding a forest.

The authors evaluate the method on four experimental regimes: (a) purely Gaussian synthetic data, (b) purely discrete synthetic data, (c) synthetic data with a mixture of Gaussian and discrete variables, and (d) real‑world biomedical datasets containing both measurement types. Metrics include structural recovery accuracy (the proportion of correctly identified edges), overall log‑likelihood, and final MDL value. Across all settings, the generalized algorithm outperforms the original Chow‑Liu and Suzuki methods. In mixed‑type scenarios, the baseline either over‑connects (inflating model complexity) or under‑connects (losing true dependencies), whereas the proposed approach consistently identifies the correct forest structure while achieving lower MDL scores, indicating a better balance of fit and parsimony.

The significance of the work lies in two dimensions. First, it provides a theoretically sound and computationally tractable way to learn probabilistic graphical models from heterogeneous data without preprocessing steps such as discretization or separate modeling of continuous variables. Second, by preserving the MDL‑based forest selection, the method yields interpretable models that are robust to overfitting, a crucial property for domains like genomics or clinical decision support where model transparency is essential.

Future research directions suggested include extending the mutual‑information estimator to non‑Gaussian continuous distributions (e.g., using kernel density estimation or copula models), incorporating sparsity‑inducing priors within a Bayesian MDL framework, and developing parallel or distributed implementations to handle high‑dimensional, large‑scale datasets. Such extensions would further broaden the applicability of the generalized Chow‑Liu approach in modern data‑intensive scientific and engineering problems.