A new embedding quality assessment method for manifold learning

Manifold learning is a hot research topic in the field of computer science. A crucial issue with current manifold learning methods is that they lack a natural quantitative measure to assess the quality of learned embeddings, which greatly limits their applications to real-world problems. In this paper, a new embedding quality assessment method for manifold learning, named as Normalization Independent Embedding Quality Assessment (NIEQA), is proposed. Compared with current assessment methods which are limited to isometric embeddings, the NIEQA method has a much larger application range due to two features. First, it is based on a new measure which can effectively evaluate how well local neighborhood geometry is preserved under normalization, hence it can be applied to both isometric and normalized embeddings. Second, it can provide both local and global evaluations to output an overall assessment. Therefore, NIEQA can serve as a natural tool in model selection and evaluation tasks for manifold learning. Experimental results on benchmark data sets validate the effectiveness of the proposed method.

💡 Research Summary

Manifold learning has become a cornerstone technique for extracting low‑dimensional representations from high‑dimensional data, yet the community still lacks a universal, quantitative metric to judge the quality of the learned embeddings. Existing measures—such as the Procrustes Measure, Local Continuity‑Monotonicity, LCMC, and Residual Variance—are either based on preserving Euclidean distance ranks or on rigid‑body alignment. Consequently, they work well for isometric methods (e.g., ISOMAP, RML) but fail dramatically when the embedding is normalized, i.e., when each coordinate is scaled to unit variance or when an anisotropic scaling is applied (as in LLE, LE, MVU, LTSA). In normalized embeddings the distance ranking within local neighborhoods changes, and rigid‑body alignment cannot compensate for coordinate‑wise scaling, leading to misleadingly high error scores.

The authors address this gap by introducing two tightly coupled contributions. First, they propose the Anisotropic Scaling Independent Measure (ASIM). For each data point i they consider its k‑nearest‑neighbor set X_i in the original space and the corresponding low‑dimensional set Y_i. They assume that X_i can be obtained from Y_i by a combination of (i) a rigid motion (rotation P_i and translation t_i) and (ii) an anisotropic diagonal scaling D_i. Formally, x_{ij}=P_i D_i y_{ij}+t_i for j=1,…,k. The optimal (P_i^, D_i^, t_i^*) are obtained by solving a constrained least‑squares problem with P_i^T P_i=I_m and D_i diagonal. The ASIM score is then defined as the normalized reconstruction error

 M_asim(X_i,Y_i)=‖X_i−P_i^* D_i^* Y_i−t_i^* e^T‖_F^2 / ‖X_i‖_F^2.

Because D_i explicitly removes any coordinate‑wise scaling, M_asim remains low even when the embedding has undergone arbitrary anisotropic normalization. A synthetic 2‑D example demonstrates that traditional Procrustes error jumps to 0.8 after normalization, whereas ASIM stays near zero, confirming its robustness.

The second contribution is the Normalization Independent Embedding Quality Assessment (NIEQA) framework, which aggregates ASIM scores into a comprehensive quality metric. NIEQA consists of:

1. Local Score – the average ASIM over all data points, reflecting how well each local neighborhood is preserved after accounting for rotation, translation, and anisotropic scaling.

2. Global Score – a similar ASIM computed on a set of landmark points that capture the manifold’s skeleton, thereby measuring preservation of the global geometric structure.

By reporting both scores, NIEQA provides a nuanced view: a method may preserve local neighborhoods well but distort the overall shape, or vice versa. The authors argue that these two complementary components make NIEQA suitable for model selection, hyper‑parameter tuning, and comparative benchmarking across different manifold learning algorithms.

Experimental validation is thorough. The authors test five representative algorithms—ISOMAP (isometric), LLE, HLLE, MVU, and LTSA (all normalized)—on synthetic manifolds (S‑curve, Swiss Roll) and real datasets (ORL face images, USPS handwritten digits). For each algorithm they compute traditional metrics (Procrustes, LCMC, Residual Variance) and the proposed NIEQA scores. Results show that for normalized methods the traditional metrics either over‑penalize (high error) or become unstable, while NIEQA consistently yields low local and global scores that correlate with visual inspection of the embeddings. Moreover, when varying the neighborhood size k, NIEQA exhibits smooth, monotonic behavior, indicating robustness to this critical hyper‑parameter. The authors also demonstrate a practical use case: by scanning a grid of LLE and MVU parameters and selecting the configuration with the minimal NIEQA score, they automatically obtain embeddings that are both locally faithful and globally coherent, eliminating the need for subjective visual selection.

In summary, the paper makes a significant methodological contribution by explicitly modeling and removing anisotropic scaling effects in the assessment of manifold embeddings. ASIM provides a mathematically sound, closed‑form similarity measure that is invariant to rotation, translation, and per‑coordinate scaling. Building on ASIM, NIEQA delivers a dual‑level (local + global) quality index that is applicable to both isometric and normalized embeddings. The extensive experiments confirm that NIEQA outperforms existing measures in terms of stability, interpretability, and relevance to downstream tasks such as model selection. Future work suggested includes extending ASIM to time‑varying manifolds, integrating it with supervised downstream objectives, and developing faster optimization schemes for very large datasets.