Algorithms for nonnegative matrix factorization with the beta-divergence

This paper describes algorithms for nonnegative matrix factorization (NMF) with the beta-divergence (beta-NMF). The beta-divergence is a family of cost functions parametrized by a single shape parameter beta that takes the Euclidean distance, the Kullback-Leibler divergence and the Itakura-Saito divergence as special cases (beta = 2,1,0, respectively). The proposed algorithms are based on a surrogate auxiliary function (a local majorization of the criterion function). We first describe a majorization-minimization (MM) algorithm that leads to multiplicative updates, which differ from standard heuristic multiplicative updates by a beta-dependent power exponent. The monotonicity of the heuristic algorithm can however be proven for beta in (0,1) using the proposed auxiliary function. Then we introduce the concept of majorization-equalization (ME) algorithm which produces updates that move along constant level sets of the auxiliary function and lead to larger steps than MM. Simulations on synthetic and real data illustrate the faster convergence of the ME approach. The paper also describes how the proposed algorithms can be adapted to two common variants of NMF : penalized NMF (i.e., when a penalty function of the factors is added to the criterion function) and convex-NMF (when the dictionary is assumed to belong to a known subspace).

💡 Research Summary

The paper “Algorithms for nonnegative matrix factorization with the beta‑divergence” introduces a unified framework for non‑negative matrix factorization (NMF) that is based on the β‑divergence, a family of cost functions that includes the Euclidean distance (β = 2), the Kullback‑Leibler (KL) divergence (β = 1) and the Itakura‑Saito (IS) divergence (β = 0) as special cases. By treating β as a continuous shape parameter, the authors obtain a flexible loss that can be tuned to the statistical characteristics of the data (e.g., Gaussian, Poisson or multiplicative Gamma noise).

The central technical contribution is the derivation of two families of multiplicative update rules that are guaranteed to decrease the β‑divergence objective while respecting the non‑negativity constraints. The first family follows a classic majorization‑minimization (MM) scheme. The authors construct a surrogate auxiliary function that locally upper‑bounds the true cost, and then minimize this bound analytically. The resulting update for each factor (W or H) has the form

  W ← W ⊙