Universal Conceptual Structure in Neural Translation: Probing NLLB-200's Multilingual Geometry

Do neural machine translation models learn language-universal conceptual representations, or do they merely cluster languages by surface similarity? We investigate this question by probing the representation geometry of Meta’s NLLB-200, a 200-language encoder-decoder Transformer, through six experiments that bridge NLP interpretability with cognitive science theories of multilingual lexical organization. Using the Swadesh core vocabulary list embedded across 135 languages, we find that the model’s embedding distances significantly correlate with phylogenetic distances from the Automated Similarity Judgment Program ($ρ= 0.13$, $p = 0.020$), demonstrating that NLLB-200 has implicitly learned the genealogical structure of human languages. We show that frequently colexified concept pairs from the CLICS database exhibit significantly higher embedding similarity than non-colexified pairs ($U = 42656$, $p = 1.33 \times 10^{-11}$, $d = 0.96$), indicating that the model has internalized universal conceptual associations. Per-language mean-centering of embeddings improves the between-concept to within-concept distance ratio by a factor of 1.19, providing geometric evidence for a language-neutral conceptual store analogous to the anterior temporal lobe hub identified in bilingual neuroimaging. Semantic offset vectors between fundamental concept pairs (e.g., man to woman, big to small) show high cross-lingual consistency (mean cosine = 0.84), suggesting that second-order relational structure is preserved across typologically diverse languages. We release InterpretCognates, an open-source interactive toolkit for exploring these phenomena, alongside a fully reproducible analysis pipeline.

💡 Research Summary

The paper “Universal Conceptual Structure in Neural Translation: Probing NLLB‑200’s Multilingual Geometry” investigates whether a massive multilingual neural machine translation (NMT) system, Meta’s NLLB‑200, learns language‑independent conceptual representations or merely clusters languages based on surface similarity. To answer this, the authors conduct six complementary experiments that blend NLP interpretability techniques with cognitive‑science theories of multilingual lexical organization.

First, they construct a multilingual lexical probe using the Swadesh core vocabulary (101 basic concepts) translated into 135 languages supported by NLLB‑200. Each target word is placed inside a minimal English carrier sentence (“I saw a {word} near the river”) and fed to the model; the encoder hidden state corresponding to the target token is extracted from the final transformer layer. This contextual embedding strategy avoids the dominance of positional and start‑of‑sequence tokens that would arise from isolated word inputs. As a control, they also extract “bare‑word” embeddings (no surrounding context) to verify that the main findings are not an artifact of the carrier sentence.

Raw embeddings from large language models are known to be highly anisotropic, inflating cosine similarity scores. The authors therefore apply a two‑stage correction: (1) All‑But‑The‑Top (ABTT) isotropy correction, subtracting the global mean and projecting out the top three principal components, which removes dominant frequency‑ and language‑identity directions; (2) per‑language mean‑centering, where each language’s centroid (mean over all concepts) is subtracted before any distance or PCA analysis. This isolates the language‑neutral subspace that should contain the shared conceptual geometry.

The six experiments are as follows:

1. Swadesh Convergence Ranking – For each concept, the mean pairwise cosine similarity across all language pairs is computed, yielding a convergence score. Concepts with high scores (e.g., “water”, “person”) are encoded uniformly across languages, whereas low‑scoring concepts show greater cross‑lingual dispersion, reflecting cultural or environmental variation.

2. Phylogenetic Correlation – An embedding‑based language‑by‑language distance matrix (averaged over all Swadesh items) is compared to the genetic distance matrix from the ASJP database using a Mantel test with 999 permutations. The result (ρ = 0.13, p = 0.020) demonstrates a statistically significant, albeit modest, alignment between the model’s geometry and known language family relationships, indicating that NLLB‑200 implicitly captures genealogical structure.

3. Colexification Sensitivity – Using the CLICS‑3 database, the authors identify concept pairs that are colexified (lexified by the same word form) in many languages. They test whether such pairs have higher embedding similarity than non‑colexified pairs via a Mann‑Whitney U test. The test yields U = 42656, p = 1.33 × 10⁻¹¹, Cohen’s d = 0.96, confirming that the model’s space respects universal semantic proximities reflected in natural colexification patterns.

4. Conceptual Store Structure (Mean‑Centering Effect) – The ratio of between‑concept to within‑concept distances is computed before and after per‑language mean‑centering. After centering, the ratio improves by a factor of 1.19, providing geometric evidence for a language‑neutral “conceptual store” overlaid with language‑specific offsets, mirroring the anterior temporal lobe hub hypothesis from bilingual neuroimaging.

5. Offset Invariance (Semantic Vector Consistency) – For 22 fundamental concept pairs (e.g., man→woman, big→small, fire→water), the authors calculate the difference vectors (offsets) in each language’s embedding space and then measure cross‑lingual cosine similarity of these vectors. The mean cosine similarity of 0.84 indicates that relational structure (the direction of meaning change) is highly preserved across typologically diverse languages, supporting the idea of an isomorphic semantic space.

6. Additional Validation (Color Terms & Loan‑Word Controls) – The authors replicate the analysis on universal color term geometry (Berlin‑Kay hierarchy) and conduct regression analyses to ensure that surface‑form similarity (e.g., loanwords) accounts for less than 2 % of the observed convergence, reinforcing that the effects are genuinely semantic rather than orthographic.

Methodologically, the paper excels in its multi‑layered validation: contextual versus bare‑word embeddings, sensitivity analysis of the ABTT hyperparameter k, and regression controls for surface similarity. The use of non‑parametric statistical tests (Mantel, Mann‑Whitney, effect size reporting) adds rigor. Moreover, the authors release an open‑source toolkit, InterpretCognates, and a fully reproducible pipeline, enhancing transparency and facilitating future work.

Limitations are acknowledged. The carrier sentence is English‑based, potentially introducing syntactic bias for languages with radically different word order or morphology. The choice of k = 3 for ABTT is fixed across all languages, whereas anisotropy may vary per language; adaptive selection could improve isotropy correction. Only 135 of the 200 supported languages are examined, leaving low‑resource languages under‑represented. The offset‑invariance analysis uses a relatively small set of concept pairs; expanding to richer relational categories (verb‑noun, attribute‑object) would provide a more comprehensive picture.

In sum, the study provides compelling evidence that a large‑scale multilingual translation model internalizes a language‑neutral conceptual space that aligns with linguistic phylogeny, colexification patterns, and universal semantic relations. By linking model geometry to cognitive‑neuroscientific theories (e.g., the anterior temporal lobe hub), the work bridges NLP and cognitive science, suggesting that NMT models can serve as computational testbeds for hypotheses about the human multilingual mind. Future directions include extending the probe to more typologically extreme languages, incorporating additional relational dimensions, and directly comparing model representations with neuroimaging data to assess the correspondence between artificial and biological conceptual stores.