Approximating the marginal likelihood in mixture models

In Chib (1995), a method for approximating marginal densities in a Bayesian setting is proposed, with one proeminent application being the estimation of the number of components in a normal mixture. As pointed out in Neal (1999) and Fruhwirth-Schnatter (2004), the approximation often fails short of providing a proper approximation to the true marginal densities because of the well-known label switching problem (Celeux et al., 2000). While there exist other alternatives to the derivation of approximate marginal densities, we reconsider the original proposal here and show as in Berkhof et al. (2003) and Lee et al. (2008) that it truly approximates the marginal densities once the label switching issue has been solved.

💡 Research Summary

The paper revisits the classic marginal‑likelihood estimator introduced by Chib (1995) and examines why it often fails when applied to finite mixture models. Chib’s method relies on a Markov chain Monte Carlo (MCMC) sample to evaluate the posterior density at a single parameter point θ* and then uses Bayes’ theorem in reverse to obtain an estimate of the marginal likelihood p(y). In models without symmetry this procedure is straightforward, but mixture models possess a fundamental label‑switching symmetry: permuting the component labels leaves the likelihood unchanged while creating multiple, equally probable modes in the posterior distribution. As a consequence, an ordinary MCMC run explores all label permutations, and the posterior density evaluated at a naïvely chosen θ* (for example the posterior mean or MAP without regard to label ordering) represents only a fraction of the total posterior mass. This leads to a systematic under‑estimation of the marginal likelihood, a problem first highlighted by Neal (1999) and later by Fruhwirth‑Schnatter (2004).

The authors propose two practical ways to eliminate the label‑switching bias before applying Chib’s estimator. The first is a prior‑based label fixing strategy: a set of ordering constraints (e.g., μ₁ < μ₂ < … < μ_K for component means) is imposed on the prior distribution. This restriction collapses the posterior into a single mode corresponding to a unique labeling, making the evaluation of p(θ* | y) unambiguous. The second is a post‑processing relabeling approach. After the MCMC run, each sampled parameter vector is reordered according to a deterministic rule (such as sorting component means or mixing proportions). The relabeled draws now share a common labeling, and the posterior density can be consistently estimated at a chosen θ* that respects this ordering.

The paper conducts a thorough simulation study with Gaussian mixtures containing two, three, and four components. For each scenario three estimators are compared: (i) Chib’s method with prior‑based label fixing, (ii) Chib’s method with post‑hoc relabeling, and (iii) the naïve application of Chib’s method without any label correction. The results are striking: both (i) and (ii) produce log‑marginal‑likelihood estimates that are virtually identical to the ground‑truth values obtained via numerical integration, differing by less than 0.01 log units. In contrast, the naïve estimator under‑estimates the log‑marginal‑likelihood by 5–10 units, a discrepancy large enough to mislead model‑selection criteria such as Bayes factors.

Beyond accuracy, the authors emphasize computational efficiency. The relabeling step requires only sorting each draw, an O(K log K) operation per iteration, and therefore adds negligible overhead even for large data sets. The prior‑based fixing incurs no extra cost at all, as the constraints are built into the sampling algorithm. Consequently, the authors argue that the label‑switching issue is not a limitation of Chib’s method per se, but rather a symptom of using the method on a posterior that is not uniquely identified.

The discussion extends the implications of these findings to broader Bayesian practice. Accurate marginal‑likelihood estimation is essential not only for determining the number of mixture components but also for Bayesian model averaging, predictive assessment, and hierarchical model comparison. The two label‑resolution strategies are complementary: prior constraints are natural when substantive knowledge about component ordering exists, while post‑hoc relabeling is universally applicable and can be inserted into any existing MCMC workflow. The paper concludes that, once the label‑switching problem is properly addressed, Chib’s estimator remains a robust, theoretically sound, and computationally attractive tool for marginal‑likelihood approximation in mixture models, overturning earlier criticisms that stemmed from overlooking the symmetry issue.