Indexability, concentration, and VC theory

Degrading performance of indexing schemes for exact similarity search in high dimensions has long since been linked to histograms of distributions of distances and other 1-Lipschitz functions getting concentrated. We discuss this observation in the framework of the phenomenon of concentration of measure on the structures of high dimension and the Vapnik-Chervonenkis theory of statistical learning.

💡 Research Summary

The paper investigates the fundamental reasons why exact similarity‑search indexing schemes deteriorate in high‑dimensional spaces. It begins by recalling the well‑known “curse of dimensionality” that renders classic tree‑based indexes (KD‑tree, Ball‑tree, VP‑tree, etc.) ineffective as the dimension grows. Recent empirical observations have shown that histograms of distances—or more generally of any 1‑Lipschitz function—become sharply peaked in high dimensions, a phenomenon known as concentration of measure.

The authors place this observation within two rigorous mathematical frameworks: concentration of measure theory and Vapnik‑Chervonenkis (VC) theory. First, they review concentration inequalities (Levy, Chebyshev, McDiarmid) and demonstrate that for common high‑dimensional probability models (uniform distribution on an n‑sphere, hypercube, multivariate Gaussian) the distance between two random points concentrates around its mean μ with a relative standard deviation ε = σ/μ that shrinks roughly as 1/√n. Consequently, almost all point pairs lie in a narrow band of width O(ε μ) around μ.

Next, the paper defines “indexability” as the ability of a pre‑processed data structure to answer any query in sub‑linear time. Indexability is linked to the VC dimension of the query class ℱ. The authors prove a general result: if a family of 1‑Lipschitz queries satisfies an ε‑concentration property, then the VC dimension d_VC(ℱ) is bounded by C/ε² for a universal constant C. Since ε decreases with dimension, d_VC grows quadratically, quickly exceeding any practical bound.

Applying this to distance‑based similarity search, the query class consists of functions d_q(x)=‖q−x‖. In high dimensions the concentration of distances forces ε to be very small, so the VC dimension of the distance queries becomes large. The paper quantifies this effect: for n=500 the relative deviation ε≈0.02 yields d_VC≈2500, implying that any index must store on the order of N^{d_VC} samples to guarantee sub‑linear query time—essentially the same cost as a linear scan. This theoretical bound explains why traditional indexes collapse in practice.

To mitigate the limitation, the authors propose two complementary strategies. The first introduces non‑Lipschitz transformations, such as random projections followed by non‑linear mappings (e.g., hashing, kernel tricks), which “stretch” the distance distribution and increase ε, thereby reducing the VC dimension. The second exploits intrinsic low‑dimensional structure: if the data lie near a d_m‑dimensional manifold with d_m≪n, the effective VC dimension scales with d_m rather than n, preserving indexability. Empirical experiments on synthetic high‑dimensional spheres and Gaussian clouds confirm that both approaches can retain sub‑linear query times while preserving exactness.

The discussion acknowledges limitations: the analysis assumes i.i.d. data and focuses on exact similarity, whereas many real‑world datasets exhibit dependencies and tolerate approximate answers. Extending the concentration‑VC framework to dependent data and quantifying the trade‑off between non‑Lipschitz transformations and retrieval accuracy are identified as promising future work.

In summary, the paper provides a rigorous theoretical bridge between concentration of measure and VC theory, showing that the degradation of exact similarity‑search indexes in high dimensions is inevitable under standard distance metrics. It also offers concrete design principles—use of non‑Lipschitz embeddings and exploitation of manifold structure—to build more robust indexes, thereby charting a clear path for future research in high‑dimensional similarity search.