Semantic Sort: A Supervised Approach to Personalized Semantic Relatedness

We propose and study a novel supervised approach to learning statistical semantic relatedness models from subjectively annotated training examples. The proposed semantic model consists of parameterized co-occurrence statistics associated with textual units of a large background knowledge corpus. We present an efficient algorithm for learning such semantic models from a training sample of relatedness preferences. Our method is corpus independent and can essentially rely on any sufficiently large (unstructured) collection of coherent texts. Moreover, the approach facilitates the fitting of semantic models for specific users or groups of users. We present the results of extensive range of experiments from small to large scale, indicating that the proposed method is effective and competitive with the state-of-the-art.

💡 Research Summary

Semantic Sort introduces a supervised framework for learning statistical semantic relatedness models that can be personalized to individual users or groups. The core idea is to treat a large, unstructured text collection (e.g., Wikipedia, Project Gutenberg) as background knowledge and to extract parameterized co‑occurrence statistics for every pair of lexical items. These statistics are then fitted to human‑provided relative preference data, where annotators indicate that one word‑pair is more related than another. By casting the problem as a binary classification or ranking task, the authors employ an empirical risk minimization (ERM) objective that penalizes mismatches between the model’s predicted score differences and the observed preferences. The loss can be instantiated as a hinge or logistic loss, and the model parameters (the weighted co‑occurrence scores) are optimized with stochastic gradient methods. Only a few hyper‑parameters—regularization strength and learning rate—are required, making the approach lightweight and easy to tune.

A key contribution is the explicit handling of subjectivity in semantic relatedness. Traditional unsupervised or hand‑crafted methods (e.g., WordNet‑based information‑content measures, Wikipedia link‑based scores, raw co‑occurrence vectors) produce a single “universal” relatedness score that cannot capture personal biases such as cultural background, domain expertise, or temporal context. Semantic Sort solves this by allowing the same underlying statistical model to be re‑trained or fine‑tuned on a user’s own preference judgments, thereby yielding a personalized scoring function without changing the underlying corpus or model architecture.

The authors evaluate the method on several standard benchmark datasets: Rubenstein‑Goodenough (65 pairs), Miller‑Charles (30 pairs), and WordSim‑353 (353 pairs). They compare against a wide range of baselines, including classic lexical measures (Resnik, Jiang‑Conrath, Banerjee‑Pedersen), graph‑based approaches (Personalized PageRank on WordNet), and recent distributional embeddings (Word2Vec, GloVe). Experiments are conducted using two distinct background corpora: an older snapshot of Wikipedia and the full text of Project Gutenberg books. Results show that the base Semantic Sort model already matches or exceeds most unsupervised baselines; for example, on WordSim‑353 it achieves a Spearman ρ of 0.71, surpassing many WordNet‑based methods. When user‑specific preference data (as little as 20 % of the training pairs) are incorporated, the correlation improves by 0.05–0.08 on average, demonstrating the practical benefit of personalization.

Efficiency is another strength. The co‑occurrence matrix is stored sparsely, and training on a vocabulary of roughly 100 k terms converges within 30 minutes on a standard workstation, using less than 2 GB of RAM. This makes the approach scalable to larger vocabularies and suitable for on‑the‑fly personalization in real‑world systems.

The paper also discusses limitations. Relative preference labeling, while cheaper than absolute scoring, still requires human effort, and the model’s performance degrades for very rare word pairs due to data sparsity. Moreover, because the method relies solely on word‑level co‑occurrence, it does not capture contextual nuances needed for polysemous words or sentence‑level semantics. The authors suggest future extensions such as Bayesian smoothing for rare pairs, integration with contextual embeddings (e.g., BERT), and multi‑modal extensions that incorporate visual or structured knowledge.

In summary, Semantic Sort offers a practical, corpus‑independent, and easily personalized solution for semantic relatedness. It bridges the gap between data‑driven statistical modeling and the inherently subjective nature of human semantic judgments, achieving competitive performance on established benchmarks while remaining computationally lightweight. The work opens avenues for personalized language understanding in applications ranging from targeted advertising to adaptive information retrieval and beyond.