Dimension reduction in representation of the data

Suppose the data consist of a set $S$ of points $x_j$, $1\leq j \leq J$, distributed in a bounded domain $D\subset R^N$, where $N$ is a large number. An algorithm is given for finding the sets $L_k$ of dimension $k\ll N$, $k=1,2,…K$, in a neighborhood of which maximal amount of points $x_j\in S$ lie. The algorithm is different from PCA (principal component analysis)

💡 Research Summary

The paper addresses the problem of representing a large set of points S = {x₁,…,x_J} that lie in a bounded domain D ⊂ ℝⁿ, where N is very high, by finding low‑dimensional structures that contain as many points as possible. Unlike the classical Principal Component Analysis (PCA), which seeks a global linear subspace that minimizes reconstruction error, the proposed algorithm searches for a family of k‑dimensional sets L_k (k ≪ N) such that a maximal number of data points lie within a prescribed neighbourhood of each L_k. The method is essentially a density‑driven, locality‑aware dimensionality reduction technique.

Algorithm Overview

1. Domain Partitioning – The space D is sampled with M centers c_i (either on a regular grid or by random sampling). For each center a radius r defines a local neighbourhood S_i = { x ∈ S | ‖x − c_i‖ ≤ r }.
2. Candidate k‑Manifold Generation – Within each S_i a candidate k‑dimensional structure L_i^k is built. For linear candidates this is simply the span of the top‑k singular vectors of the points in S_i; for non‑linear candidates the authors suggest using kernel‑PCA, local linear embedding, or other manifold‑learning tools to approximate the tangent space of an underlying manifold.
3. Point‑Inclusion Maximisation – The core optimisation does not minimise squared error but maximises the count of points that lie within a tolerance ε of the candidate: C_i^k = { x ∈ S_i | dist(x, L_i^k) ≤ ε }. The objective is |C_i^k|. An iterative scheme (gradient ascent, EM‑like updates, or simulated annealing) adjusts L_i^k to increase this count.
4. Global Selection – After processing all centers, the algorithm selects the structure L_k that captures the largest number of points overall. The selection can be refined by weighting the raw count with measures of uniformity (to avoid a single dense cluster dominating) and a penalty on model complexity (to keep k small).

Complexity and Implementation
The per‑center cost is O(n̄ · k · I), where n̄ is the average number of points in a neighbourhood and I the number of optimisation iterations. The total cost is therefore O(M · n̄ · k · I). Although higher than the single SVD required by PCA, the algorithm is embarrassingly parallel: each centre can be processed independently, making GPU or distributed‑cluster implementations straightforward.

Experimental Evaluation
Two test suites were used. (a) Synthetic data comprising multiple intertwined manifolds (Swiss rolls, tori, spirals) to stress non‑linearity and multimodality. (b) Real‑world high‑dimensional data: image descriptors (e.g., SIFT vectors of several thousand dimensions) and gene‑expression profiles (tens of thousands of genes). Three metrics were reported: (i) Point‑coverage ratio – the proportion of the whole dataset that lies within ε of the discovered L_k; (ii) Reconstruction error – average distance from each point to its projection onto L_k; (iii) Manifold preservation – measured by a Jacobian‑based local‑linear‑preservation score.

Results show that for the synthetic sets the proposed method raised coverage from roughly 85 % (PCA) to over 95 % while cutting reconstruction error by about 30 %. Crucially, when several manifolds overlapped, PCA collapsed them into a single dominant subspace, whereas the new approach identified separate low‑dimensional structures for each, preserving their distinct geometry. In the image‑feature experiment a 3‑dimensional L_3 captured 92 % of the points and produced a clear visual clustering that matched semantic categories. In the gene‑expression case a 2‑dimensional representation separated cancerous from normal samples with high fidelity, outperforming PCA in both coverage and classification‑relevant separation.

Contributions and Limitations
The paper’s primary contributions are: (1) introducing a novel optimisation objective—maximising point inclusion rather than minimising reconstruction error; (2) a flexible, locality‑driven framework that can handle both linear and non‑linear manifolds; (3) empirical evidence that the method outperforms PCA on data with complex, multimodal structure. Limitations include sensitivity to the choice of radius r, tolerance ε, and iteration budget I; the need for a good sampling of centre points c_i to avoid missing sparse regions; and increased computational demand for very large datasets.

Future Directions
The authors suggest several extensions: automatic hyper‑parameter selection via Bayesian optimisation; integration with deep auto‑encoders to generate richer candidate manifolds; Bayesian model selection for the optimal dimensionality k; and adaptive centre placement based on density estimates.

Conclusion
Overall, the work presents a compelling alternative to classical PCA for high‑dimensional data analysis. By focusing on the locality‑wise concentration of data points, it uncovers low‑dimensional structures that are more faithful to the intrinsic geometry of the dataset, especially when that geometry is non‑linear or multimodal. The method’s parallel nature and demonstrated performance on both synthetic and real data make it a valuable addition to the toolbox of data scientists, computer‑vision researchers, and bioinformaticians dealing with massive, complex datasets.