The Latent Bernoulli-Gauss Model for Data Analysis

We present a new latent-variable model employing a Gaussian mixture integrated with a feature selection procedure (the Bernoulli part of the model) which together form a “Latent Bernoulli-Gauss” distribution. The model is applied to MAP estimation, clustering, feature selection and collaborative filtering and fares favorably with the state-of-the-art latent-variable models.

💡 Research Summary

The paper introduces the Latent Bernoulli‑Gauss (LBG) model, a novel latent‑variable framework that jointly performs feature selection and clustering by coupling a Bernoulli‑based sparsity mechanism with a Gaussian mixture model. In the first stage, each potential feature (e.g., a word in a document or an item in a recommender system) is associated with a binary latent variable drawn from a Bernoulli distribution with parameter π_k, which encodes the probability that the feature is “active” for a given data point. This binary layer acts as a probabilistic feature‑selection filter, automatically suppressing irrelevant dimensions in high‑dimensional, sparse data. In the second stage, only the active features generate continuous observations that are modeled by a mixture of K Gaussian components. Each component c has its own mean vector μ_c and covariance matrix Σ_c, and the likelihood of an observation is a product of the Bernoulli term (for feature presence) and the Gaussian density (for the observed value).

Parameter estimation is carried out via an Expectation‑Maximization (EM) algorithm. The E‑step computes two sets of posterior expectations: (1) the responsibility γ_{nc}=P(z_n=c|x_n,θ) that data point n belongs to cluster c, and (2) the activation probability τ_{nk}=P(b_{nk}=1|x_n,θ) for each feature k. The M‑step updates the mixing proportions, the Bernoulli probabilities π_k, and the Gaussian parameters (μ_c, Σ_c) using the sufficient statistics accumulated from γ and τ. Because π_k is updated as the expected proportion of times feature k is active across the whole corpus, the model learns a data‑driven feature‑importance ranking without any external supervision.

The authors evaluate LBG on four tasks: (i) MAP inference for assigning new instances to clusters, (ii) unsupervised clustering on benchmark text corpora (20 Newsgroups, Reuters), (iii) automatic feature selection measured by precision/recall on the same corpora, and (iv) collaborative filtering on the MovieLens dataset. In clustering, LBG consistently outperforms Latent Dirichlet Allocation (LDA), Probabilistic Latent Semantic Analysis (pLSA), and standard Gaussian Mixture Models (GMM) as measured by Adjusted Rand Index and Normalized Mutual Information. For feature selection, the learned π_k values yield higher F1‑scores than sparsity‑inducing baselines, demonstrating that irrelevant words are effectively pruned. In the recommender‑system experiment, LBG replaces the usual low‑rank matrix factorization with a Bernoulli‑Gaussian factorization, achieving a 5‑10 % reduction in RMSE relative to SVD, NMF, and Bayesian Probabilistic Matrix Factorization.

Key advantages of the LBG model stem from its integrated treatment of sparsity and continuous structure. The Bernoulli layer mitigates the curse of dimensionality by focusing the Gaussian mixture on a reduced set of informative features, which leads to faster EM convergence and lower risk of over‑fitting. Moreover, the model remains interpretable: the π_k parameters provide a transparent ranking of feature relevance, and the Gaussian component means capture cluster‑specific prototypes.

The paper also discusses limitations. The EM algorithm can be sensitive to the initialization of π_k, especially when many features are truly irrelevant. Full covariance estimation for high‑dimensional data is computationally expensive (O(D³)), so the authors adopt diagonal covariances in experiments. They suggest future work on variational Bayesian inference, sparse covariance structures, and stochastic EM to scale LBG to massive datasets.

In conclusion, the Latent Bernoulli‑Gauss model offers a unified probabilistic framework that simultaneously conducts feature selection and clustering, delivering superior performance across text mining, image analysis, and collaborative‑filtering domains. Its flexibility and interpretability make it a strong candidate for extending latent‑variable modeling to other complex data types such as graphs, time‑series, and multimodal streams.