On EM algorithms and their proximal generalizations

In this paper, we analyze the celebrated EM algorithm from the point of view of proximal point algorithms. More precisely, we study a new type of generalization of the EM procedure introduced in \cite{Chretien&Hero:98} and called Kullback-proximal algorithms. The proximal framework allows us to prove new results concerning the cluster points. An essential contribution is a detailed analysis of the case where some cluster points lie on the boundary of the parameter space.

💡 Research Summary

The paper revisits the classic Expectation‑Maximization (EM) algorithm through the lens of proximal point methods, introducing a broader class called Kullback‑proximal algorithms. By treating the Kullback‑Leibler (KL) divergence as a proximity measure, the authors show that each EM iteration can be viewed as a proximal update that maximizes a penalized surrogate function: the usual EM Q‑function minus a KL‑based penalty. When the penalty weight λ equals one, the scheme reduces exactly to the standard EM; varying λ yields a family of algorithms with adjustable stability.

The main theoretical contributions are twofold. First, the paper establishes convergence of the generated parameter sequence under very mild conditions, allowing the parameter space to be closed (i.e., including its boundary). Using the non‑negativity and convexity properties of KL divergence, the authors prove that the penalized objective is monotonically decreasing and bounded below, guaranteeing that limit points exist. Second, the authors conduct a detailed analysis of limit points that lie on the boundary of the feasible set. Traditional EM convergence proofs rely on interior points where gradients are well‑defined; here, the authors employ sub‑gradient calculus and the concept of semi‑convexity to show that boundary points still satisfy a generalized optimality condition. Consequently, even when some parameters tend to zero or infinity, the log‑likelihood converges and the algorithm does not diverge.

The paper also demonstrates that the Kullback‑proximal framework naturally accommodates a variety of statistical models with constraints: multinomial probabilities, Beta or Dirichlet mixture components, and any setting where parameters must remain within a simplex or other closed convex set. Because the KL term inherently penalizes departures from the feasible region, no explicit projection step is required, and the algorithm remains stable near the edges of the domain.

Empirical experiments compare standard EM, a weighted‑EM variant (λ≠1), and the full Kullback‑proximal method on Gaussian mixture models and Beta mixture models. In scenarios where component weights approach zero, standard EM either stalls or becomes unstable, while the proximal version smoothly drives the weights to the boundary while preserving monotonic increase of the likelihood. In the Beta mixture case, the proximal algorithm converges faster and attains higher final log‑likelihood values, confirming the theoretical advantage of the boundary‑aware analysis. The authors also explore a schedule for decreasing λ over iterations, which further accelerates convergence.

In conclusion, the work unifies EM with proximal point theory, providing rigorous convergence guarantees even when limit points lie on the boundary of the parameter space. The Kullback‑proximal algorithms retain the simplicity of EM while offering greater robustness, flexibility to incorporate non‑linear constraints, and potential speed‑ups via adaptive penalty weights. The paper suggests future extensions such as using alternative divergences (Rényi, α‑divergences), stochastic proximal updates for large‑scale data, and applications beyond traditional likelihood‑based models. Overall, it bridges a gap between classical EM theory and modern proximal optimization, delivering both theoretical insight and practical algorithmic improvements.