UMAP Is Spectral Clustering on the Fuzzy Nearest-Neighbor Graph

UMAP (Uniform Manifold Approximation and Projection) is among the most widely used algorithms for non linear dimensionality reduction and data visualisation. Despite its popularity, and despite being presented through the lens of algebraic topology, the exact relationship between UMAP and classical spectral methods has remained informal. In this work, we prove that UMAP performs spectral clustering on the fuzzy k nearest neighbour graph. Our proof proceeds in three steps: (1) we show that UMAP’s stochastic optimisation with negative sampling is a contrastive learning objective on the similarity graph; (2) we invoke the result of HaoChen et al. [8], establishing that contrastive learning on a similarity graph is equivalent to spectral clustering; and (3) we verify that UMAP’s spectral initialisation computes the exact linear solution to this spectral problem. The equivalence is exact for Gaussian kernels, and holds as a first order approximation for UMAP’s default Cauchy type kernel. Our result unifies UMAP, contrastive learning, and spectral clustering under a single framework, and provides theoretical grounding for several empirical observations about UMAP’s behaviour.

💡 Research Summary

The paper establishes a rigorous theoretical link between UMAP (Uniform Manifold Approximation and Projection) and classical spectral clustering. While UMAP is usually presented through the language of fuzzy simplicial sets and algebraic topology, its exact relationship to spectral methods has remained informal. The authors prove that UMAP is, in fact, performing spectral clustering on the fuzzy k‑nearest‑neighbor (k‑NN) graph. The proof proceeds in three logical steps.

First, they reinterpret UMAP’s stochastic optimization with negative sampling as a contrastive learning objective. UMAP minimizes a fuzzy cross‑entropy loss (Equation 5) by repeatedly sampling a positive edge (a,b) with probability proportional to the fuzzy weight v_{ab} and n_{neg} negative nodes uniformly at random. The per‑step loss ℓ(a)=−log Φ(y_a,y_b)−∑{i=1}^{n{neg}}log(1−Φ(y_a,y_{c_i})) is exactly the Noise‑Contrastive Estimation (NCE) formulation used in many contrastive self‑supervised methods (e.g., SimCLR, InfoNCE). The positive term pulls neighboring points together, while the negative term pushes random points apart, matching the canonical structure of contrastive objectives.

Second, they invoke the result of Hao‑Chen et al. (2022) which shows that contrastive learning with the InfoNCE loss is mathematically equivalent to spectral clustering on the underlying similarity graph. By expanding the attractive term for two kernel choices they obtain:

• Gaussian kernel Φ_G(y_i,y_j)=exp(−‖y_i−y_j‖²/2τ) → L_attract = (1/τ) tr(Yᵀ L(V) Y), an exact Laplacian quadratic form.

• Default Cauchy‑type kernel Φ(y_i,y_j)=(1+a‖y_i−y_j‖²)^{−b} with b=1. For neighboring pairs where a‖y_i−y_j‖²≪1, a first‑order Taylor expansion yields L_attract ≈ 2a tr(Yᵀ L(V) Y).

Thus, the attractive component of UMAP’s loss is (up to a constant) the standard spectral clustering objective tr(Zᵀ L(W) Z). The repulsive term L_repel, derived from the (1−v_{ij}) log(1−Φ) part of the loss, plays the same role as the log‑partition function in InfoNCE: it prevents collapse by penalising embeddings that assign high similarity to non‑neighbors. Although the functional forms differ (binary cross‑entropy per edge versus softmax over all pairs), both act as barrier functions ensuring a non‑degenerate solution.

Third, they demonstrate that UMAP’s spectral initialization is exactly the solution of the normalized Laplacian eigenproblem. UMAP constructs the fuzzy adjacency matrix V, forms the normalized Laplacian ˜L(V)=D^{−1/2}(D−V)D^{−1/2}, and takes the bottom d non‑zero eigenvectors as the initial embedding Y₀. This is precisely the minimizer of min_{ZᵀZ=I} tr(Zᵀ ˜L(V) Z), i.e., the linear spectral clustering solution (also known as Laplacian Eigenmaps).

Putting these pieces together, the full UMAP pipeline can be expressed as:

1. Build the fuzzy k‑NN similarity matrix V.
2. Initialise Y₀ with the eigenvectors of the normalized Laplacian ˜L(V).
3. Refine Y via stochastic gradient descent on a contrastive loss that decomposes into a Laplacian quadratic term plus a repulsive regulariser.

Consequently, UMAP is exactly spectral clustering on V, with the SGD phase providing a non‑linear, kernel‑dependent refinement of the linear solution. The equivalence is exact for Gaussian kernels and holds as a first‑order approximation for the default Cauchy kernel.

The paper also discusses practical implications. The choice of k directly shapes the graph topology and thus the spectrum of L(V); larger k yields smoother eigenvectors (global structure), while smaller k captures finer local detail. Kernel choice affects only the refinement stage: Gaussian kernels give a purely linear objective, whereas the heavy‑tailed Cauchy kernel mitigates the crowding problem without altering the underlying clustering. The number of negative samples controls the strength of the repulsive term, analogous to regularisation in the spectral objective. Early stopping preserves more of the global structure encoded in the eigenvectors, while many SGD epochs push the embedding toward a more locally faithful configuration.

In summary, the authors provide a complete theoretical unification of three seemingly disparate strands: (i) UMAP’s fuzzy‑set cross‑entropy optimisation, (ii) contrastive self‑supervised learning, and (iii) classical spectral clustering. All three minimise a cross‑entropy divergence between a fixed similarity graph and a learned embedding graph, and the dominant term in each case is the Laplacian quadratic form. The only substantive difference lies in how the similarity graph is constructed: UMAP uses adaptive bandwidth k‑NN distances, contrastive learning uses augmentation‑induced affinities, and traditional spectral clustering employs a fixed kernel. This unified view explains many empirical observations about UMAP’s behaviour, justifies the importance of spectral initialisation, and offers a principled framework for tuning UMAP’s hyper‑parameters.