Approximate Maximum A Posteriori Inference with Entropic Priors

In certain applications it is useful to fit multinomial distributions to observed data with a penalty term that encourages sparsity. For example, in probabilistic latent audio source decomposition one may wish to encode the assumption that only a few latent sources are active at any given time. The standard heuristic of applying an L1 penalty is not an option when fitting the parameters to a multinomial distribution, which are constrained to sum to 1. An alternative is to use a penalty term that encourages low-entropy solutions, which corresponds to maximum a posteriori (MAP) parameter estimation with an entropic prior. The lack of conjugacy between the entropic prior and the multinomial distribution complicates this approach. In this report I propose a simple iterative algorithm for MAP estimation of multinomial distributions with sparsity-inducing entropic priors.

💡 Research Summary

The paper addresses the problem of estimating the parameters of a multinomial distribution when a sparsity-inducing prior is desired. A naïve L1‑penalty cannot be used because the multinomial parameters must sum to one, so the author proposes an alternative prior that favours low‑entropy solutions. Specifically, the prior is defined as

 p(θ) ∝ exp { a ∑ₖ θₖ log θₖ }

with a > 0, which assigns higher probability to distributions that concentrate most of their mass on a few categories. This “entropic prior” is not conjugate to the multinomial likelihood, making the MAP estimate analytically intractable.

To overcome this difficulty, the author introduces an auxiliary probability vector α (∑ₖ αₖ = 1, αₖ ≥ 0) and a scalar ν > 1, and defines a surrogate objective

 ℓ(a, ν, θ, α) = a ∑ₖ αₖ