Distributed Representations of Words and Phrases and their Compositionality

The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of “Canada” and “Air” cannot be easily combined to obtain “Air Canada”. Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible.

💡 Research Summary

The paper “Distributed Representations of Words and Phrases and their Compositionality” builds on the continuous Skip‑gram model, a neural network architecture that learns word embeddings by predicting surrounding words given a target word. While the original Skip‑gram model already demonstrated impressive scalability—training on over 100 billion words in a single day—it suffered from three practical limitations: (1) excessive computation on very frequent function words, (2) reliance on a hierarchical softmax layer that still required a log‑size tree traversal for each training example, and (3) an inability to represent multi‑word idiomatic phrases whose meaning is not a simple composition of their parts.

To address these issues the authors introduce three complementary techniques. First, subsampling of frequent words discards each word w with probability P(w)=1−√(t/f(w)), where f(w) is its corpus frequency and t≈10⁻⁵. This dramatically reduces the number of training updates involving high‑frequency tokens such as “the” or “of”, yielding a 2‑10× speed‑up and, more importantly, allowing the model to allocate more learning capacity to rarer, more informative words.

Second, they replace the hierarchical softmax with Negative Sampling (NEG), a simplified version of Noise‑Contrastive Estimation. For each positive (target, context) pair the model draws k negative samples from a noise distribution Pₙ(w) proportional to the unigram frequency raised to the 3/4 power. The objective becomes
  log σ(v′ₒ·vᵢ) + ∑{j=1}^{k} E{w_j∼Pₙ}