The Corpus Replication Task

In the field of Natural Language Processing (NLP), we revisit the well-known word embedding algorithm word2vec. Word embeddings identify words by vectors such that the words’ distributional similarity is captured. Unexpectedly, besides semantic similarity even relational similarity has been shown to be captured in word embeddings generated by word2vec, whence two questions arise. Firstly, which kind of relations are representable in continuous space and secondly, how are relations built. In order to tackle these questions we propose a bottom-up point of view. We call generating input text for which word2vec outputs target relations solving the Corpus Replication Task. Deeming generalizations of this approach to any set of relations possible, we expect solving of the Corpus Replication Task to provide partial answers to the questions.

💡 Research Summary

The paper revisits the widely used word2vec algorithm and asks a fundamental question: which kinds of relational information can be represented in continuous vector space, and how does word2vec actually build those relations? To answer this, the authors introduce a bottom‑up, reverse‑engineering approach they call the “Corpus Replication Task”. The idea is simple yet powerful: construct an artificial corpus T such that, when fed to word2vec, the resulting embeddings satisfy a target relation R (e.g., king – man ≈ queen – woman or Germany + capital ≈ Berlin). If the corpus produces the desired relation, it is said to “solve” R.

The paper first reviews the distributional hypothesis and distinguishes paradigmatic similarity (words sharing the same contexts, typical of synonyms) from syntagmatic similarity (words co‑occurring together, covering both semantic and syntactic patterns). It notes that while word2vec captures paradigmatic similarity well, its performance on syntagmatic similarity plateaus around 65 % in standard benchmarks. The authors argue that the limitation may stem from the noisy, uncontrolled nature of natural text, and that a noise‑free, deliberately crafted corpus could yield higher syntagmatic accuracy.

Two experiments are presented. In the first, a syntactic analogy is targeted: (king – man) ≈ (queen – woman). The authors define two base sentences, “A king is a man.” and “A queen is a woman.”, and concatenate them according to a Bernoulli distribution with p = 0.5. Using the Skip‑Gram version of word2vec, a window size n = 2, and a two‑dimensional embedding space, they observe that the vectors for “king” and “queen” collapse together, as do “man” and “woman”. Consequently, the difference vectors become near zero, effectively solving the relation. The result demonstrates that when two word pairs share identical left‑right contexts, word2vec treats them as paradigmatically similar, leading to the desired linear relationship.

The second experiment tackles a semantic analogy: (Germany + capital) ≈ Berlin. Three base sentences are used: “Berlin is the capital of Germany.”, “Germany has a capital.”, and “Berlin is the capital.”. The sentences are sampled uniformly and concatenated to form corpora of increasing size: 1 000, 10 000, and 100 000 copies. With the same window size (n = 2) and 2‑D embeddings, the authors find that the relation does not hold for the smallest corpus; the vector sum of “Germany” and “capital” is far from “Berlin”. However, as the corpus grows to 10 000 and 100 000 copies, the Euclidean distance between the sum and “Berlin” shrinks dramatically, and the relation is effectively satisfied. This scaling experiment shows that sufficient co‑occurrence statistics are required for word2vec to learn meaningful semantic compositions.

A third set of observations concerns the impact of the window size. The authors note that n = 2 is optimal for the chosen sentences because it aligns the target words within each other’s context windows. Reducing the window to n = 1 reduces context overlap, causing vectors to disperse; increasing it to n = 3 makes all words share too many contexts, collapsing the vectors into a single point. Thus, the choice of window size directly controls the degree of contextual independence among word groups.

The paper also discusses dimensionality. All experiments are performed in two dimensions, which limits the number of mutually orthogonal context groups that can be represented. The authors argue that in three or higher dimensions, one can embed multiple independent context clusters without forcing them to share the same direction, suggesting a path toward scaling the approach to richer vocabularies and more complex relational sets.

Finally, the authors test non‑uniform sampling of base sentences (e.g., very low probability p for one sentence) and find that the learned relations are robust: even with p as low as 0.002, the vectors still converge appropriately, though the rate of convergence varies. They acknowledge that more intricate probability distributions, especially those with strong conditional dependencies, remain an open research direction.

In conclusion, the study demonstrates that artificially generated, noise‑free corpora can be deliberately engineered to make word2vec learn specific relational patterns, both syntactic and semantic. By controlling sentence templates, sampling probabilities, and the context window, one can replicate classic analogy tests in a controlled setting. The authors view this “Corpus Replication Task” as a stepping stone toward deeper understanding of how distributional information translates into linear regularities in word embeddings, and they outline future work on higher‑dimensional embeddings, complex corpora, and systematic analysis of the limits of relation replication.