Improving Interpretability of Lexical Semantic Change with Neurobiological Features

Lexical Semantic Change (LSC) is the phenomenon in which the meaning of a word change over time. Most studies on LSC focus on improving the performance of estimating the degree of LSC, however, it is often difficult to interpret how the meaning of a word change. Enhancing the interpretability of LSC is a significant challenge as it could lead to novel insights in this field. To tackle this challenge, we propose a method to map the semantic space of contextualized embeddings of words obtained by a pre-trained language model to a neurobiological feature space. In the neurobiological feature space, each dimension corresponds to a primitive feature of words, and its value represents the intensity of that feature. This enables humans to interpret LSC systematically. When employed for the estimation of the degree of LSC, our method demonstrates superior performance in comparison to the majority of the previous methods. In addition, given the high interpretability of the proposed method, several analyses on LSC are carried out. The results demonstrate that our method not only discovers interesting types of LSC that have been overlooked in previous studies but also effectively searches for words with specific types of LSC.

💡 Research Summary

The paper addresses a long‑standing problem in lexical semantic change (LSC) research: while many methods can accurately quantify how much a word’s meaning has shifted over time, they offer little insight into how that meaning has changed. To make LSC interpretable, the authors propose a two‑step framework that maps contextualized word embeddings from a pre‑trained BERT model into a neurobiological feature space originally introduced by Binder et al. (2016). This “Binder” space consists of 65 dimensions (e.g., Vision, Audition, Happy) whose values are derived from brain‑based semantic representations and are publicly available for 535 English words.

Mapping BERT to Binder Space
For each target word, the authors average BERT‑base‑uncased hidden states across all its occurrences in a historical corpus (CCOHA) to obtain a 768‑dimensional vector r_w. They then train a regression function ψ that predicts the 65‑dimensional Binder vector b_w from r_w. Two architectures are evaluated: a simple linear transformation (LT) and a four‑layer multilayer perceptron (MLP) with ReLU activations and a final sigmoid scaled to the 0‑6 range of Binder scores. Training uses mean‑squared error on the 535 labeled words, with 10‑fold cross‑validation. The linear model consistently yields lower MSE (≈0.57) than the MLP (≈0.64), suggesting that the relationship between BERT embeddings and neurobiological features is largely linear.

Detecting LSC with the Mapped Space
To estimate the degree of change between two time periods (t₁ and t₂), the authors compute Binder vectors for each occurrence of a word in each period, then calculate the average pairwise distance (APD) between the two sets. They experiment with Euclidean, cosine, and Spearman distance; cosine proves the most stable. The method is evaluated on SemEval‑2020 Task 1, which provides human‑annotated LSC scores for 37 English words (periods 1810‑1860 vs. 1960‑2010). Using the linear mapping and cosine distance, the system achieves a Spearman correlation of 0.667, outperforming all baseline approaches that rely solely on the raw BERT space (≈0.616) and ranking among the best among methods that do not incorporate external lexical resources. While external‑knowledge‑enhanced models (e.g., XL‑LEXEME, 0.757) still lead, the proposed approach offers a compelling trade‑off: competitive performance with a transparent, cognitively grounded interpretation.

Analyzing Types of Semantic Change
Beyond scoring change magnitude, the authors exploit the interpretability of the Binder dimensions to characterize how meanings evolve. For each word, they compute an LSC vector v_lsc(w) = mean(U_{t₂}^w) – mean(U_{t₁}^w), where each component reflects the gain or loss of a specific neurobiological feature. Selecting the 500 words with the largest LSC‑vector norms (i.e., strongest changes), they apply Sparse Principal Component Analysis (Sparse PCA) with ten components. Sparse PCA forces many loadings to zero, making each component easy to interpret as a cluster of a few Binder features. The top three features per component are reported, and the authors manually inspect representative words whose scores on a component are extreme. For example, a component dominated by Vision, Motion, and Spatial captures the shift of “plane” from a geometric concept to an aircraft; another component emphasizing Positive, Arousal, and Valence reflects the amelioration of “terrific.”

To validate these interpretations, the authors assign pseudo‑senses to word usages via k‑means clustering of contextual embeddings, examine exemplar sentences, and verify that the distribution of senses changes in a manner consistent with the identified Binder feature shifts. This pipeline mirrors earlier PCA‑based analyses in raw embedding space but gains a far clearer semantic narrative thanks to the neurobiological grounding.

Detecting Specific Change Types: Amelioration and Pejoration
The framework also enables targeted searches for words that have become more positive (amelioration) or more negative (pejoration) over time. By focusing on the Valence and Arousal dimensions of the LSC vectors, the authors automatically retrieve candidate words and confirm that many align with known historical shifts, while also surfacing previously undocumented cases.

Contributions and Limitations

1. Introduces neurobiological Binder features as an interpretable semantic axis for LSC.
2. Demonstrates that a simple linear mapping from BERT to Binder space preserves enough information for accurate LSC scoring.
3. Shows that Sparse PCA on LSC vectors can uncover coherent, cognitively meaningful types of semantic change.
4. Provides a method for automatically detecting amelioration and pejoration.

Limitations include reliance on a modest set of 535 labeled words, which may restrict generalization to low‑frequency or domain‑specific vocabulary, and the current focus on English. Extending Binder‑style annotations to other languages or leveraging larger brain‑imaging datasets could broaden applicability. Moreover, while linear mapping works well here, future work might explore whether richer, possibly multimodal, mappings could capture subtler aspects of meaning change.

Overall Assessment
The paper makes a significant interdisciplinary contribution by bridging NLP, cognitive neuroscience, and historical linguistics. It offers a practical, interpretable alternative to black‑box LSC detectors, and its empirical results—both in quantitative prediction and qualitative analysis—support the claim that neurobiological feature spaces can serve as a transparent lens for studying how word meanings evolve across centuries.