Bayesian orthogonal component analysis for sparse representation

This paper addresses the problem of identifying a lower dimensional space where observed data can be sparsely represented. This under-complete dictionary learning task can be formulated as a blind separation problem of sparse sources linearly mixed with an unknown orthogonal mixing matrix. This issue is formulated in a Bayesian framework. First, the unknown sparse sources are modeled as Bernoulli-Gaussian processes. To promote sparsity, a weighted mixture of an atom at zero and a Gaussian distribution is proposed as prior distribution for the unobserved sources. A non-informative prior distribution defined on an appropriate Stiefel manifold is elected for the mixing matrix. The Bayesian inference on the unknown parameters is conducted using a Markov chain Monte Carlo (MCMC) method. A partially collapsed Gibbs sampler is designed to generate samples asymptotically distributed according to the joint posterior distribution of the unknown model parameters and hyperparameters. These samples are then used to approximate the joint maximum a posteriori estimator of the sources and mixing matrix. Simulations conducted on synthetic data are reported to illustrate the performance of the method for recovering sparse representations. An application to sparse coding on under-complete dictionary is finally investigated.

💡 Research Summary

The paper tackles the problem of finding a low‑dimensional subspace in which observed data admit a sparse representation, a scenario commonly referred to as under‑complete dictionary learning. Unlike the usual over‑complete setting, the dictionary (mixing matrix) has fewer columns than rows (K < N), which makes the identification of both the dictionary and the sparse codes more challenging. The authors cast the problem as a blind source separation task with an unknown orthogonal mixing matrix A and sparse source matrix S, expressed as X = A S + E, where E is Gaussian noise.

To enforce sparsity, each source coefficient s_{kt} is modeled by a Bernoulli‑Gaussian (BG) prior: a mixture of a point mass at zero and a zero‑mean Gaussian with variance σ². The mixing weight π (the probability of being “active”) directly controls the level of sparsity; small π yields a highly sparse representation. For the mixing matrix, a non‑informative uniform prior is placed on the Stiefel manifold V_{K,N}, i.e., the set of N × K matrices with orthonormal columns. This prior automatically guarantees the orthogonality constraint without introducing extra hyper‑parameters. Hyper‑parameters π and σ² are given conjugate Beta and inverse‑Gamma priors, respectively, allowing them to be learned from the data.

Exact Bayesian inference is intractable, so the authors develop a Markov chain Monte Carlo (MCMC) algorithm based on a partially collapsed Gibbs sampler. The key idea is to sample the pair (S, π) jointly, effectively integrating out π when updating S, which reduces the strong posterior coupling and accelerates mixing. Sampling of A is performed directly on the Stiefel manifold using Householder reflections or Cayley transforms, thereby preserving orthogonality without resorting to Metropolis‑Hastings corrections. The noise variance σ² is updated via its inverse‑Gamma full conditional, exploiting conjugacy. After a sufficient burn‑in period, the generated samples approximate the joint posterior p(A, S, π, σ² | X). The authors then compute either posterior means or the joint maximum a posteriori (MAP) estimate, the latter corresponding to the solution of a constrained optimization problem that simultaneously enforces orthogonality of A and sparsity of S.

Experimental validation is carried out on synthetic data and on image‑patch coding tasks. In the synthetic experiments, dimensions N = 30, K = 10, and varying sparsity levels (π = 0.05–0.2) and signal‑to‑noise ratios (10–30 dB) are considered. The proposed Bayesian method consistently outperforms conventional orthogonal matching pursuit (OMP) and K‑SVD‑based under‑complete approaches, achieving 15–30 % lower reconstruction error and higher support‑recovery F‑scores. For image patches (8 × 8 vectors, N = 64, K = 32), the method yields an average PSNR gain of about 2.1 dB over baseline techniques, with visibly reduced artifacts. Moreover, the MCMC samples provide posterior variances for each coefficient, enabling a confidence map that highlights uncertain entries—a feature useful for downstream tasks such as adaptive denoising or model‑based decision making.

The paper’s contributions are threefold: (1) a principled Bayesian formulation that explicitly incorporates an orthogonal mixing matrix via a Stiefel‑manifold prior; (2) a sparsity‑inducing Bernoulli‑Gaussian prior that offers a tunable and interpretable sparsity level; and (3) an efficient partially collapsed Gibbs sampler that respects the manifold constraints and mitigates posterior coupling. While the approach demonstrates superior performance, the authors acknowledge the computational burden of MCMC, especially for large‑scale problems, and suggest future work on variational approximations or stochastic gradient MCMC to improve scalability. They also note that relaxing the strict orthogonality assumption could broaden applicability to scenarios where near‑orthogonal dictionaries are more realistic. In summary, the study provides a solid Bayesian framework for under‑complete sparse coding, delivering both accurate reconstructions and valuable uncertainty quantification, and opens avenues for further research on efficient inference and more flexible dictionary models.