Feature Space Topology Control via Hopkins Loss

Feature space topology refers to the organization of samples within the feature space. Modifying this topology can be beneficial in machine learning applications, including dimensionality reduction, generative modeling, transfer learning, and robustness to adversarial attacks. This paper introduces a novel loss function, Hopkins loss, which leverages the Hopkins statistic to enforce a desired feature space topology, which is in contrast to existing topology-related methods that aim to preserve input feature topology. We evaluate the effectiveness of Hopkins loss on speech, text, and image data in two scenarios: classification and dimensionality reduction using nonlinear bottleneck autoencoders. Our experiments show that integrating Hopkins loss into classification or dimensionality reduction has only a small impact on classification performance while providing the benefit of modifying feature topology.

💡 Research Summary

The paper introduces a novel loss function called “Hopkins loss,” which leverages the Hopkins statistic to actively shape the topology of a learned feature space. Unlike prior work that seeks to preserve the input data’s topological structure, Hopkins loss allows a practitioner to prescribe a desired arrangement—regularly spaced, randomly spaced, or clustered—and to drive the network toward that configuration during training.

The Hopkins statistic H quantifies clustering tendency: values near 0.5 indicate a random (Poisson) distribution, values below ~0.3 signal a regular (uniform) layout, and values above ~0.7 denote strong clustering. The authors make H differentiable by computing nearest‑neighbor distances within each mini‑batch, using Chebyshev (L∞) distance because it consistently preserves H’s behavior across dimensions and dataset sizes. For a batch X of size n, a small subset ˜X (size m ≪ n) is sampled, and a synthetic set Y of m uniformly random points is generated within the same range as X. The statistic is computed as

 H = Σ_i u_i / (Σ_i u_i + Σ_i w_i)

where u_i is the distance from a synthetic point y_i to its nearest neighbor in X, and w_i is the distance from a real sampled point ˜x_i to its nearest neighbor in X. The loss is defined as L_H = |H – H_T|, with H_T being a user‑defined target (e.g., 0.01 for regular spacing, 0.5 for randomness, 0.99 for clustering).

Training combines Hopkins loss with a primary objective: for classification, L = w_C·L_CE + (1–w_C)·L_H; for autoencoding, L = w_R·L_MSE + (1–w_R)·L_H. The authors empirically set w_C = 0.75 and w_R = 0.75, giving the topology term a substantial but not dominant influence.

Experiments span three modalities—speech emotion recognition (RA‑VDESS, 88‑dim eGeMAPS features), text sentiment classification (IMDB, 768‑dim BERT CLS embeddings), and fashion image classification (Fashion‑MNIST, 784‑dim pixel vectors). For each modality, they evaluate (1) pure classification performance and (2) a bottleneck autoencoder with latent dimensions B = 32, 8, and 2.

Key findings:

1. Classification Impact – Adding Hopkins loss (any H_T) changes overall accuracy by less than 0.5 % relative to a baseline trained with only cross‑entropy loss. This holds across all three datasets, indicating that the topology regularizer does not substantially harm discriminative ability.

2. Topology Control – Measured H values in the learned representations closely track the target H_T. For H_T = 0.01 the average H ≈ 0.85–0.90, for H_T = 0.5 the average H ≈ 0.70–0.73, and for H_T = 0.99 the average H ≈ 0.80–0.99. Thus the loss reliably drives the feature space toward the prescribed arrangement.

3. Autoencoder Compression – When the autoencoder is trained with Hopkins loss, the compressed latent space inherits the desired topology without dramatically degrading reconstruction quality. Linear classifiers built on the bottleneck features achieve comparable accuracy to baselines, especially at higher latent dimensions (B = 32). Even at B = 2, the regular (H_T = 0.01) and clustered (H_T = 0.99) configurations produce distinct, visually separable point clouds, suggesting utility for visualization and downstream tasks that benefit from structured embeddings.

4. Distance Metric Choice – The authors tested Euclidean, Manhattan, cosine, Mahalanobis, and Chebyshev distances. Only Chebyshev distance consistently preserved the intended H behavior across dimensions (d = 2 to 784) and dataset sizes (n = 1 k to 100 k). This choice is critical for practical deployment.

5. Sampling Ratio – Following prior work, the authors set the number of sampled points m = 0.05 n, finding little sensitivity to variations between 0.02 n and 0.08 n.

The paper’s contributions are threefold: (i) a mathematically simple, differentiable formulation of a topology‑shaping loss, (ii) empirical evidence that the loss can be combined with standard objectives without harming performance, and (iii) demonstration across speech, text, and image domains that the method can enforce regular, random, or clustered feature arrangements.

Limitations noted include the reliance on Chebyshev distance (potentially less suitable for certain data manifolds), the extra computational overhead of nearest‑neighbor calculations per mini‑batch, and the need for further study on how prescribed topologies affect robustness to adversarial perturbations or transfer learning scenarios. Future work could integrate Hopkins loss into generative adversarial networks, variational autoencoders, or domain‑adaptation pipelines to align source and target feature spaces, and could explore adaptive target H_T values that evolve during training.

Overall, the study opens a new avenue for actively sculpting feature spaces, complementing existing topology‑preserving techniques and offering a flexible tool for researchers seeking structured embeddings in diverse machine‑learning applications.