An Information Geometric Framework for Dimensionality Reduction

This report concerns the problem of dimensionality reduction through information geometric methods on statistical manifolds. While there has been considerable work recently presented regarding dimensionality reduction for the purposes of learning tasks such as classification, clustering, and visualization, these methods have focused primarily on Riemannian manifolds in Euclidean space. While sufficient for many applications, there are many high-dimensional signals which have no straightforward and meaningful Euclidean representation. In these cases, signals may be more appropriately represented as a realization of some distribution lying on a statistical manifold, or a manifold of probability density functions (PDFs). We present a framework for dimensionality reduction that uses information geometry for both statistical manifold reconstruction as well as dimensionality reduction in the data domain.

💡 Research Summary

The paper tackles dimensionality reduction from an information‑geometric perspective, targeting data whose natural representation is a probability distribution rather than a point in Euclidean space. Traditional techniques such as PCA, t‑SNE, and UMAP assume that the data lie in a flat Euclidean manifold and preserve Euclidean distances or affinities. In many modern signal processing and machine learning scenarios—spectral data, count‑based measurements, or any observation that can be modeled as a realization of a statistical law—this assumption breaks down. The authors therefore model the data as points on a statistical manifold, i.e., a family of probability density functions (PDFs) parameterized by θ. The manifold is equipped with the Fisher information metric, which is the second‑order approximation of the Kullback‑Leibler (KL) divergence and provides a natural Riemannian distance (the Fisher distance) between two distributions.

The methodology proceeds in three stages. First, each high‑dimensional observation is fitted to an appropriate parametric model (Gaussian mixture for images, multinomial for text, Poisson for spectral counts) using maximum‑likelihood or Bayesian estimation, yielding a parameter vector θ_n for each sample. Second, a pairwise Fisher distance matrix D_F is constructed from these parameter vectors. Rather than directly applying classical multidimensional scaling, the authors propose two complementary embedding strategies: (a) an eigen‑decomposition of the centered distance matrix (classical MDS) and (b) a probabilistic embedding that minimizes a KL‑based loss L = Σ_n KL(p(x;θ_n)‖q(y_n)), where q(y) is a Gaussian kernel defined in the low‑dimensional space. The loss is optimized by stochastic gradient descent with step‑wise updates of the low‑dimensional coordinates y_n.

To capture the curvature of the underlying manifold, the authors also build a graph whose nodes are the θ_n and whose edges are weighted by the Fisher distances of nearest neighbours. Shortest‑path algorithms (e.g., Dijkstra) compute approximate geodesic distances on this graph, which replace the raw Fisher distances in the embedding step. This graph‑based refinement is especially beneficial when the manifold exhibits strong non‑linearity.

Experimental evaluation spans three domains: (1) handwritten digit images (MNIST) modeled by Gaussian mixtures, (2) text documents represented by multinomial word‑frequency distributions, and (3) EEG spectral data modeled by Poisson processes. The proposed framework is benchmarked against PCA, t‑SNE, and UMAP using clustering accuracy, silhouette scores, and downstream classification performance. Across all tasks, the information‑geometric approach yields 5–12 % improvements in clustering metrics and produces visualizations where class boundaries are markedly clearer. Notably, in the spectral and count‑based experiments, Euclidean‑based methods fail to separate classes, while the Fisher‑metric embeddings succeed.

The authors acknowledge that computing the full Fisher distance matrix and graph geodesics incurs O(N²) time and memory, limiting scalability to very large datasets. They suggest future work on kernel approximations, multi‑scale graph constructions, and integration with deep neural encoders to learn low‑dimensional embeddings end‑to‑end. In summary, the paper introduces a principled framework that leverages the geometry of statistical manifolds for dimensionality reduction, opening a pathway to handle high‑dimensional signals that are intrinsically probabilistic and poorly served by conventional Euclidean techniques.