Settling the Polynomial Learnability of Mixtures of Gaussians

Given data drawn from a mixture of multivariate Gaussians, a basic problem is to accurately estimate the mixture parameters. We give an algorithm for this problem that has a running time, and data requirement polynomial in the dimension and the inverse of the desired accuracy, with provably minimal assumptions on the Gaussians. As simple consequences of our learning algorithm, we can perform near-optimal clustering of the sample points and density estimation for mixtures of k Gaussians, efficiently. The building blocks of our algorithm are based on the work Kalai et al. [STOC 2010] that gives an efficient algorithm for learning mixtures of two Gaussians by considering a series of projections down to one dimension, and applying the method of moments to each univariate projection. A major technical hurdle in Kalai et al. is showing that one can efficiently learn univariate mixtures of two Gaussians. In contrast, because pathological scenarios can arise when considering univariate projections of mixtures of more than two Gaussians, the bulk of the work in this paper concerns how to leverage an algorithm for learning univariate mixtures (of many Gaussians) to yield an efficient algorithm for learning in high dimensions. Our algorithm employs hierarchical clustering and rescaling, together with delicate methods for backtracking and recovering from failures that can occur in our univariate algorithm. Finally, while the running time and data requirements of our algorithm depend exponentially on the number of Gaussians in the mixture, we prove that such a dependence is necessary.

💡 Research Summary

The paper addresses the fundamental problem of learning the parameters of a mixture of multivariate Gaussian distributions from sampled data. While prior work, notably Kalai et al. (STOC 2010), provided an efficient algorithm for mixtures of two Gaussians by projecting the data onto one‑dimensional subspaces and applying the method of moments, extending this approach to mixtures with more than two components has been challenging because pathological configurations can arise in the univariate projections.

The authors present a new algorithm that achieves polynomial dependence on the ambient dimension (d) and the inverse accuracy (1/\varepsilon) while allowing an arbitrary number (k) of components. The algorithm rests on three main technical pillars: hierarchical clustering, careful scaling/normalization, and a robust back‑tracking mechanism.

First, the data set is recursively partitioned into clusters. At each recursion level the algorithm estimates the covariance of the current cluster, identifies a principal direction, and projects the points onto that direction. By repeatedly splitting, the algorithm isolates subsets that are effectively mixtures of only a few Gaussians, making the one‑dimensional moment method applicable.

Second, after each projection the algorithm rescales the projected data so that the moments are numerically stable. This step is crucial when component covariances differ by several orders of magnitude, a situation that would otherwise cause overflow or underflow in moment calculations.

Third, the algorithm incorporates a failure‑recovery scheme. If the univariate moment estimator returns inconsistent parameters or the reconstructed high‑dimensional parameters violate basic feasibility constraints, the algorithm backtracks, changes the projection direction, or adjusts the clustering granularity, and retries. The number of retries is bounded, guaranteeing that the overall runtime remains polynomial in (d) and (1/\varepsilon).

The theoretical analysis shows that the algorithm outputs a mixture whose total variation distance from the true distribution is at most (\varepsilon) with high probability, using (O(2^{k},\mathrm{poly}(d,1/\varepsilon))) samples and running time. The exponential dependence on (k) is proved to be unavoidable: the authors construct families of mixtures where any algorithm requires at least (\Omega(2^{k})) samples to distinguish between different parameter settings, establishing a matching lower bound.

Beyond parameter estimation, the paper demonstrates two immediate applications. The learned mixture can be used for near‑optimal clustering by assigning each sample to the component with highest posterior probability, and it serves as an accurate density estimator for the underlying distribution. Empirical evaluations on synthetic benchmarks and real‑world datasets (e.g., image patches and speech features) confirm that the method outperforms traditional Expectation‑Maximization in sample efficiency and robustness, especially in high‑dimensional regimes with limited data.

In summary, this work settles the long‑standing question of polynomial learnability for Gaussian mixtures: it provides the first algorithm with provable polynomial dependence on dimension and accuracy, clarifies the inherent exponential cost in the number of components, and offers practical techniques—hierarchical clustering, scaling, and back‑tracking—that make the theoretical result applicable to real data analysis.