Noisy Independent Factor Analysis Model for Density Estimation and Classification

We consider the problem of multivariate density estimation when the unknown density is assumed to follow a particular form of dimensionality reduction, a noisy independent factor analysis (IFA) model. In this model the data are generated by a number of latent independent components having unknown distributions and are observed in Gaussian noise. We do not assume that either the number of components or the matrix mixing the components are known. We show that the densities of this form can be estimated with a fast rate. Using the mirror averaging aggregation algorithm, we construct a density estimator which achieves a nearly parametric rate log^(1/4)n/sqrt(n), independent of the dimensionality of the data, as the sample size $n$ tends to infinity. This estimator is adaptive to the number of components, their distributions and the mixing matrix. We then apply this density estimator to construct nonparametric plug-in classifiers and show that they achieve the best obtainable rate of the excess Bayes risk, to within a logarithmic factor independent of the dimension of the data. Applications of this classifier to simulated data sets and to real data from a remote sensing experiment show promising results.

💡 Research Summary

The paper tackles the challenging problem of multivariate density estimation and subsequent classification when the underlying data-generating mechanism follows a noisy Independent Factor Analysis (IFA) model. In this model each observation X∈ℝ^d is expressed as X = A S + ε, where S = (S₁,…,S_m)ᵀ consists of m mutually independent latent components with unknown (possibly non‑Gaussian) marginal distributions, A is an unknown d × m mixing matrix, and ε is Gaussian noise with covariance σ²I. Crucially, neither the number of factors m nor the mixing matrix A is assumed to be known a priori.

Main contributions

1. Fast, dimension‑free density estimation.
   The authors construct a family of candidate estimators by (i) generating a finite set of plausible mixing matrices and factor numbers, (ii) projecting the data onto each candidate matrix to obtain one‑dimensional samples, and (iii) applying standard kernel density estimators to each projected component. The product of the one‑dimensional estimates (exploiting independence) is then mapped back to the original space, yielding a multivariate density estimate ĝ_{Â,m̂} for each candidate (Â,m̂).

2. Mirror‑averaging aggregation.
   To combine the many candidate estimators, the paper employs the mirror‑averaging (also called exponential weighting) algorithm. At each iteration the algorithm evaluates the empirical log‑likelihood loss of each candidate and updates a probability distribution over the candidates using a mirror map (the Kullback–Leibler divergence). The final aggregated estimator is a convex combination of the candidates with data‑dependent weights.

3. Theoretical risk bound.
   The authors prove that the aggregated estimator achieves an expected L₂ risk of order
   \