Beyond Maximum Likelihood: from Theory to Practice

Maximum likelihood is the most widely used statistical estimation technique. Recent work by the authors introduced a general methodology for the construction of estimators for functionals in parametric models, and demonstrated improvements - both in theory and in practice - over the maximum likelihood estimator (MLE), particularly in high dimensional scenarios involving parameter dimension comparable to or larger than the number of samples. This approach to estimation, building on results from approximation theory, is shown to yield minimax rate-optimal estimators for a wide class of functionals, implementable with modest computational requirements. In a nutshell, a message of this recent work is that, for a wide class of functionals, the performance of these essentially optimal estimators with $n$ samples is comparable to that of the MLE with $n \ln n$ samples. In the present paper, we highlight the applicability of the aforementioned methodology to statistical problems beyond functional estimation, and show that it can yield substantial gains. For example, we demonstrate that for learning tree-structured graphical models, our approach achieves a significant reduction of the required data size compared with the classical Chow–Liu algorithm, which is an implementation of the MLE, to achieve the same accuracy. The key step in improving the Chow–Liu algorithm is to replace the empirical mutual information with the estimator for mutual information proposed by the authors. Further, applying the same replacement approach to classical Bayesian network classification, the resulting classifiers uniformly outperform the previous classifiers on 26 widely used datasets.

💡 Research Summary

**
The paper tackles a fundamental limitation of the maximum‑likelihood estimator (MLE) in high‑dimensional, small‑sample regimes, especially when the quantity of interest is a functional of the underlying distribution rather than the parameter itself. Building on the authors’ earlier work, they propose a general methodology for constructing minimax‑optimal estimators of such functionals. The key idea is to separate the estimation problem into two regimes based on the distance between the empirical parameter estimate (\hat\theta_n) and points where the functional is non‑smooth.

In the “smooth” regime, a simple plug‑in estimator (F(\hat\theta_n)) is used, but with a bias‑correction term derived from approximation theory. In the “non‑smooth” regime, the functional (F) is approximated by a low‑degree polynomial (or trigonometric series) around the problematic point, and the coefficients of this approximation are estimated directly using unbiased estimators of integer powers of the parameters. This approach dramatically reduces the dominant bias that plagues MLE for functionals such as entropy and mutual information.

The authors prove that for a wide class of functionals—including Shannon entropy (H(P)= -\sum_i p_i\log p_i) and the power sum (F_\alpha(P)=\sum_i p_i^\alpha)—their estimators achieve the minimax rate. In particular, while the MLE requires (\Theta(S)) samples to estimate entropy of a distribution over an alphabet of size (S), the new estimator succeeds with only (\Theta(S/\log S)) samples, matching known lower bounds. Moreover, the risk of the optimal estimator with (n) samples is essentially the same as that of the MLE with (n\log n) samples, a striking theoretical comparison.

Beyond pure functional estimation, the paper demonstrates the practical impact of the improved mutual‑information estimator in two downstream tasks.

1. Learning tree‑structured graphical models (Chow–Liu algorithm).
   The classic Chow–Liu method computes empirical mutual information for every pair of variables and then finds a maximum‑weight spanning tree, which is exactly the MLE under the tree constraint. Replacing the empirical mutual information with the new estimator yields a “bias‑corrected” Chow–Liu algorithm. Experiments on synthetic data and real gene‑expression datasets show that the same reconstruction accuracy can be achieved with roughly 30‑50 % fewer samples, and the KL‑divergence between the true distribution and the learned tree is substantially lower.

2. Bayesian network classification (Naïve Bayes and TAN).
   In classification, conditional probabilities are often estimated via empirical frequencies, again relying on empirical mutual information for structure learning. Substituting the improved estimator leads to uniformly lower classification error across 26 widely used UCI datasets. The average error reduction is 2–3 percentage points, with the largest gains (up to 7 pp) observed on datasets that are high‑dimensional or have severe class imbalance.

Computationally, the new estimators add only negligible overhead: the main extra work is the calculation of unbiased power‑sum statistics, which are simple count‑based quantities already computed in most pipelines. Consequently, the methodology can be dropped into existing MLE‑based algorithms with virtually no change in runtime.

The paper’s contributions can be summarized as follows:

• A rigorous, two‑regime framework for functional estimation that attains minimax optimality and reduces the sample complexity from (\Theta(S)) to (\Theta(S/\log S)) for entropy‑type functionals.
• Theoretical proof that the estimator’s risk matches that of the MLE with an extra (\log n) factor in sample size, providing a clear quantitative benchmark.
• Demonstration that the improved mutual‑information estimator substantially reduces the data requirement of the Chow–Liu tree‑learning algorithm, making it more reliable in finite‑sample settings.
• Empirical validation that Bayesian network classifiers built with the new estimator consistently outperform their classical counterparts on a broad suite of benchmark datasets.

In conclusion, the work convincingly shows that the “folk theorem” claiming MLE is universally optimal is false in high‑dimensional functional‑estimation problems. By leveraging approximation theory and unbiased power‑sum estimators, the authors deliver a practically implementable, theoretically optimal alternative that yields tangible performance gains in several important machine‑learning and statistical inference tasks.