Variational Bayes approach for model aggregation in unsupervised classification with Markovian dependency

We consider a binary unsupervised classification problem where each observation is associated with an unobserved label that we want to retrieve. More precisely, we assume that there are two groups of observation: normal and abnormal. The normal' observations are coming from a known distribution whereas the distribution of the abnormal’ observations is unknown. Several models have been developed to fit this unknown distribution. In this paper, we propose an alternative based on a mixture of Gaussian distributions. The inference is done within a variational Bayesian framework and our aim is to infer the posterior probability of belonging to the class of interest. To this end, it makes no sense to estimate the mixture component number since each mixture model provides more or less relevant information to the posterior probability estimation. By computing a weighted average (named aggregated estimator) over the model collection, Bayesian Model Averaging (BMA) is one way of combining models in order to account for information provided by each model. The aim is then the estimation of the weights and the posterior probability for one specific model. In this work, we derive optimal approximations of these quantities from the variational theory and propose other approximations of the weights. To perform our method, we consider that the data are dependent (Markovian dependency) and hence we consider a Hidden Markov Model. A simulation study is carried out to evaluate the accuracy of the estimates in terms of classification. We also present an application to the analysis of public health surveillance systems.

💡 Research Summary

This paper addresses an unsupervised binary classification problem in which each observation is linked to an unobserved label indicating whether the observation belongs to a “normal” (known) or an “abnormal” (unknown) population. The normal distribution φ is assumed to be known, while the abnormal distribution f is completely unknown. To model f, the authors propose a flexible family of Gaussian mixture models with varying numbers of components. Because no single mixture model is guaranteed to be the true data‑generating process, the authors adopt a Bayesian Model Averaging (BMA) strategy: instead of selecting a single model, they combine information from a whole collection M = {f₁,…,f_M} by weighting each model according to its posterior probability given the data.

The key difficulty is that the labels S are latent, so the posterior P(M | X) involves integrating over both the hidden labels and the model parameters Θ. Direct computation is intractable. The authors therefore turn to variational Bayesian (VB) inference, which approximates the joint posterior P(S,Θ,M | X) with a tractable distribution Q(S,Θ,M) by minimizing the Kullback‑Leibler (KL) divergence. This minimization naturally decomposes into two sub‑problems: (i) optimizing the model‑weight distribution Q(M) and (ii) optimizing the conditional distribution Q(S,Θ | M). The first sub‑problem yields an expression for the optimal weights: \