Word, graph and manifold embedding from Markov processes

Continuous vector representations of words and objects appear to carry surprisingly rich semantic content. In this paper, we advance both the conceptual and theoretical understanding of word embeddings in three ways. First, we ground embeddings in semantic spaces studied in cognitive-psychometric literature and introduce new evaluation tasks. Second, in contrast to prior work, we take metric recovery as the key object of study, unify existing algorithms as consistent metric recovery methods based on co-occurrence counts from simple Markov random walks, and propose a new recovery algorithm. Third, we generalize metric recovery to graphs and manifolds, relating co-occurence counts on random walks in graphs and random processes on manifolds to the underlying metric to be recovered, thereby reconciling manifold estimation and embedding algorithms. We compare embedding algorithms across a range of tasks, from nonlinear dimensionality reduction to three semantic language tasks, including analogies, sequence completion, and classification.

💡 Research Summary

The paper advances the theory of continuous vector embeddings by grounding them in cognitive‑psychometric semantic spaces and by introducing two new inductive reasoning tasks—sequence completion and classification—alongside the classic analogy benchmark. It reframes embedding learning as a metric‑recovery problem: given co‑occurrence counts derived from simple Markov random walks over a corpus, the negative log of these counts converges to the squared Euclidean distance between latent word vectors (Lemma 1). Under this view, popular methods such as GloVe, word2vec, and PMI‑based SVD are shown to be consistent estimators of the underlying metric, differing only in weighting schemes and optimization tricks.

To test the metric‑recovery paradigm, the authors propose a novel negative‑binomial regression model that directly fits the log‑linear relationship between distances and co‑occurrences. This model yields a simple gradient update and empirically outperforms the other methods on the newly introduced tasks.

The framework is then generalized to graphs and smooth manifolds. By constructing spatial graphs whose edge probabilities depend on a kernel of the geodesic distance, the authors prove that a random walk on such a graph converges (via Skorokhod‑Varadhan theory) to a diffusion process on the manifold. Varadhan’s large‑deviation formula then links the log transition probability to the squared geodesic distance, establishing that co‑occurrence logs faithfully encode the manifold’s intrinsic metric.

Experiments on word analogy, sequence completion, classification, and nonlinear dimensionality‑reduction datasets (e.g., Swiss‑roll, MNIST‑manifold) demonstrate that the proposed regression method achieves the lowest reconstruction error on metric‑sensitive tasks while remaining competitive on analogies. Overall, the work provides a unified theoretical foundation for word, graph, and manifold embeddings, clarifying why existing algorithms work and offering a principled path toward more accurate metric recovery.