Dimensionality Reduction by Local Discriminative Gaussians

We present local discriminative Gaussian (LDG) dimensionality reduction, a supervised dimensionality reduction technique for classification. The LDG objective function is an approximation to the leave-one-out training error of a local quadratic discriminant analysis classifier, and thus acts locally to each training point in order to find a mapping where similar data can be discriminated from dissimilar data. While other state-of-the-art linear dimensionality reduction methods require gradient descent or iterative solution approaches, LDG is solved with a single eigen-decomposition. Thus, it scales better for datasets with a large number of feature dimensions or training examples. We also adapt LDG to the transfer learning setting, and show that it achieves good performance when the test data distribution differs from that of the training data.

💡 Research Summary

The paper introduces Local Discriminative Gaussian (LDG), a supervised linear dimensionality‑reduction technique that directly targets the leave‑one‑out (LOO) training error of a local quadratic discriminant analysis (QDA) classifier. For each training sample, LDG builds a local Gaussian model by estimating class‑specific means and covariances from the k‑nearest neighbors. Using these local statistics, the authors derive an approximate loss that measures how well the projected data separate the true class from the most confusing alternative class after a linear projection W∈ℝ^{D×d}. By expanding the loss to second order, the overall objective becomes a Rayleigh‑quotient‑like expression: maximize Tr(WᵀS_BW) subject to WᵀS_WW = I, where S_B aggregates between‑class scatter computed from local means and S_W aggregates within‑class scatter from local covariances. This formulation leads to a generalized eigenvalue problem; the optimal projection consists of the top d eigenvectors of S_W⁻¹S_B. Consequently, LDG can be solved with a single eigen‑decomposition, avoiding iterative gradient descent or EM procedures required by many competing methods.

Computationally, the dominant costs are (i) constructing local statistics, O(NkD), and (ii) the eigen‑decomposition, O(D³). Because the eigen‑step is performed only once, LDG scales well to very high‑dimensional data sets where N or D are large. The local modeling distinguishes LDG from global approaches such as LDA, which assume a single covariance per class and can under‑represent multimodal or nonlinear class structures. Experiments on synthetic data, handwritten digit benchmarks (MNIST, USPS, SVHN), face datasets (PIE), and object recognition collections (COIL‑20) demonstrate that LDG consistently outperforms PCA, LDA, LFDA, and other state‑of‑the‑art linear methods, especially when the target dimensionality is extremely low (e.g., 10–30 dimensions). The method also integrates seamlessly with downstream classifiers (k‑NN, linear SVM, label propagation), preserving discriminative information that is often lost by unsupervised reductions.

Beyond standard supervised settings, the authors extend LDG to transfer learning. They compute separate local statistics for source and target domains and modify the objective to penalize discrepancies between source and target within‑class covariances while still maximizing between‑class separation. This domain‑adaptive LDG is evaluated on cross‑domain benchmarks such as Office+Caltech and MNIST↔SVHN, where it achieves higher accuracy than domain‑alignment baselines like CORAL, JDA, and DAN. The adaptation works without requiring labeled target data, relying solely on the unsupervised structure captured by the local Gaussians.

In summary, LDG offers three key contributions: (1) a principled approximation of the LOO error for a locally discriminative classifier, (2) an efficient closed‑form solution via a single eigen‑decomposition, and (3) a natural extension to unsupervised domain adaptation. These properties make LDG a practical tool for high‑dimensional problems where preserving local discriminative cues is essential, and they open avenues for further research into nonlinear extensions or deep‑learning‑compatible implementations.