Improved EM for Mixture Proportions with Applications to Nonparametric ML Estimation for Censored Data

Improved EM strategies, based on the idea of efficient data augmentation (Meng and van Dyk 1997, 1998), are presented for ML estimation of mixture proportions. The resulting algorithms inherit the simplicity, ease of implementation, and monotonic convergence properties of EM, but have considerably improved speed. Because conventional EM tends to be slow when there exists a large overlap between the mixture components, we can improve the speed without sacrificing the simplicity or stability, if we can reformulate the problem so as to reduce the amount of overlap. We propose simple “squeezing” strategies for that purpose. Moreover, for high-dimensional problems, such as computing the nonparametric MLE of the distribution function with censored data, a natural and effective remedy for conventional EM is to add exchange steps (based on improved EM) between adjacent mixture components, where the overlap is most severe. Theoretical considerations show that the resulting EM-type algorithms, when carefully implemented, are globally convergent. Simulated and real data examples show dramatic improvement in speed in realistic situations.

💡 Research Summary

The paper addresses a well‑known drawback of the classical Expectation–Maximization (EM) algorithm when it is used to estimate mixture‑proportion parameters in finite mixture models: convergence can be painfully slow whenever the component densities overlap substantially. The authors build on the “efficient data augmentation” (EDA) framework introduced by Meng and van Dyk (1997, 1998) and propose two complementary strategies—“squeezing” and “exchange”—that dramatically accelerate EM while preserving its hallmark properties of monotonic likelihood increase, simplicity of implementation, and global convergence.

Squeezing.
In the standard EM formulation each observation is associated with a latent categorical label indicating from which component it originated. When components overlap, the posterior responsibilities computed in the E‑step are typically moderate values (e.g., 0.3–0.7) that change only slightly from one iteration to the next, leading to minute updates of the mixing proportions in the M‑step. The squeezing idea expands the latent space by introducing auxiliary variables that allow the responsibilities to be “compressed” toward the extremes 0 or 1. Concretely, for each observation i a squeezing factor α_i ∈ (0,1] is introduced; the augmented complete‑data log‑likelihood is then minorized by a function that depends on α_i‑scaled responsibilities. By choosing α_i close to 1 for observations that are strongly associated with a single component and smaller values for ambiguous cases, the E‑step yields a sparser responsibility matrix. Consequently the M‑step produces larger, more decisive changes in the mixing proportions, and the overall likelihood climbs much faster. The authors prove that the squeezed EM still satisfies the EM monotonicity property because the squeezing operation constitutes a valid minorization of the observed‑data log‑likelihood.

Exchange for censored data.
The second contribution tackles high‑dimensional non‑parametric maximum‑likelihood estimation (NPMLE) for censored data, a problem that can be recast as estimating a discrete distribution with K point masses (the mixture proportions). In censored settings many observation intervals intersect several point masses, creating severe overlap and rendering the ordinary EM impractically slow. The authors propose to interleave the squeezed EM iterations with “exchange steps” that act only on adjacent point masses where the overlap is most acute. An exchange step recomputes the responsibilities for a pair of neighboring components (k, k+1) and simultaneously updates both proportions by solving a small two‑component sub‑problem. Because the exchange is confined to a local pair, the computational burden remains modest, yet the local redistribution of probability mass can produce a substantial increase in the overall likelihood. By alternating squeezed E‑steps, global M‑steps, and targeted exchange updates, the algorithm retains the global convergence guarantees of EM while dramatically reducing the number of iterations required.

Theoretical guarantees.
The paper provides a rigorous convergence analysis. It shows that both squeezing and exchange steps generate valid minorizing functions for the observed‑data log‑likelihood, ensuring that each iteration does not decrease the likelihood. Moreover, the set of fixed points of the accelerated algorithm coincides with that of the original EM, implying that the accelerated method converges to the same MLE (or a stationary point in the non‑convex case). The authors also derive bounds on the rate of convergence that depend on the chosen squeezing factors and the frequency of exchange steps, illustrating that aggressive squeezing (α_i close to 1) and frequent exchanges can shrink the spectral radius of the EM operator, thereby speeding up convergence.

Empirical evaluation.
Simulation studies involve mixtures of normal distributions with varying degrees of overlap and censored survival data with interval censoring. In scenarios with high overlap, the squeezed EM alone reduces the required iterations by a factor of 5–10 relative to standard EM. When exchange steps are added for the censored NPMLE problem, the reduction becomes even more pronounced, often exceeding a factor of 20–30. Real‑world applications include a medical survival dataset with left‑, right‑, and interval‑censored observations, and an environmental monitoring dataset where pollutant concentrations are reported as detection limits. In both cases the accelerated algorithm reaches the final log‑likelihood within a few dozen iterations, whereas the conventional EM needs several hundred iterations and exhibits unstable intermediate estimates.

Computational considerations.
The algorithm’s memory footprint is comparable to that of ordinary EM because the additional squeezing parameters α_i can be stored as a vector of length n, and exchange steps require only local updates of two proportions at a time. The per‑iteration computational complexity remains O(nK), but the constant factor is reduced because the responsibility matrix becomes sparser after squeezing, and exchange steps avoid recomputing responsibilities for all components. The authors discuss practical implementation details such as adaptive selection of α_i (e.g., based on the current posterior entropy) and criteria for triggering exchange steps (e.g., when the overlap measure between neighboring components exceeds a threshold).

Conclusions and future work.
The study demonstrates that modest modifications to the latent‑variable representation—specifically, compressing ambiguous assignments and locally exchanging probability mass—can yield orders‑of‑magnitude speedups for EM without sacrificing its robustness. The techniques are broadly applicable to any mixture‑model problem where component overlap hampers convergence, and they are especially valuable for high‑dimensional non‑parametric problems such as censored‑data NPMLE. Future research directions suggested include automatic tuning of squeezing factors, parallelization of exchange updates across multiple component pairs, and extension of the framework to Bayesian EM or variational inference settings. Overall, the paper makes a significant methodological contribution by reconciling the simplicity of EM with the computational demands of modern statistical applications.