How does a Multilingual LM Handle Multiple Languages?

Multilingual language models have significantly advanced due to rapid progress in natural language processing. Models like BLOOM 1.7B, trained on diverse multilingual datasets, aim to bridge linguistic gaps. However, their effectiveness in capturing linguistic knowledge, particularly for low-resource languages, remains an open question. This study critically examines MLMs capabilities in multilingual understanding, semantic representation, and cross-lingual knowledge transfer. While these models perform well for high-resource languages, they struggle with less-represented ones. Additionally, traditional evaluation methods often overlook their internal syntactic and semantic encoding. This research addresses key limitations through three objectives. First, it assesses semantic similarity by analyzing multilingual word embeddings for consistency using cosine similarity. Second, it examines BLOOM-1.7B and Qwen2 through Named Entity Recognition and sentence similarity tasks to understand their linguistic structures. Third, it explores cross-lingual knowledge transfer by evaluating generalization from high-resource to low-resource languages in sentiment analysis and text classification. By leveraging linguistic probing, performance metrics, and visualizations, this study provides insights into the strengths and limitations of MLMs. The findings aim to enhance multilingual NLP models, ensuring better support for both high- and low-resource languages, thereby promoting inclusivity in language technologies.

💡 Research Summary

This paper presents a comprehensive investigation of multilingual language models (MLMs), focusing on BLOOM‑1.7B, Qwen‑2, BLOOM‑560M, and multilingual‑BERT‑base. The authors address three core research questions: (1) how well do multilingual word embeddings preserve semantic similarity across languages, (2) what internal representations do the models develop for cross‑lingual tasks such as sentence similarity and named‑entity recognition (NER), and (3) how effectively can knowledge learned from high‑resource languages be transferred to low‑resource languages.

In the first experiment, 5,000 English words from the Gutenberg corpus were translated into French, Spanish, German, and Chinese using Google Translate. Static embeddings generated by BLOOM‑1.7B were compared with cosine similarity. The results show high alignment (0.85‑0.92) for the three Indo‑European languages, reflecting shared lexical and syntactic properties, while Chinese exhibits a markedly lower similarity (~0.45), confirming that typological distance is reflected in the embedding space. Principal Component Analysis and t‑SNE visualizations corroborate these findings: European languages cluster tightly with English, whereas Chinese forms an isolated cluster.

The second set of experiments probes the models layer‑by‑layer. Sentence similarity is evaluated on the Opus parallel corpus for Hindi, Tamil, and Arabic; NER performance is measured on a multilingual adaptation of CoNLL‑2003. For BLOOM‑1.7B, early layers (0‑5) achieve very high similarity scores for Hindi and Tamil (0.92‑0.95) but low scores for Arabic (≈0.50). As depth increases, similarity declines for all languages, with a pronounced drop after layer 15, and hidden‑state magnitudes plunge to –1.0, indicating severe degradation of useful representations. Qwen‑2 displays a different pattern: Arabic starts lower (0.45) but quickly rises to ≈0.80, and similarity remains stable through the deeper layers (15‑25), with only minor fluctuations. Hidden‑state analysis shows that early layers produce near‑zero, stable activations, mid‑layers gradually increase to a modest positive range (0.05‑0.10), and deep layers in BLOOM suffer a sharp decline, whereas Qwen‑2 maintains healthier activations. These observations suggest that Qwen‑2’s architectural choices (e.g., attention mechanisms, layer normalization) better preserve cross‑lingual alignment in deeper layers.

The third experiment examines cross‑lingual transferability. BLOOM‑560M and multilingual‑BERT‑base are evaluated on text classification and NER for low‑resource languages Arabic and Swahili, using only English pre‑training. Baseline multilingual‑BERT shows solid transfer performance (e.g., ≈72 % F1 on Swahili NER). BLOOM‑560M initially lags but, when combined with data‑augmentation techniques (back‑translation, synonym replacement) and self‑supervised pre‑training, it gains 5‑7 percentage points on Arabic, demonstrating that larger pre‑training corpora and modern architectures can be leveraged to improve low‑resource performance.

The authors synthesize three key insights: (1) typological similarity strongly influences embedding alignment, (2) early transformer layers capture language‑agnostic semantics while deeper layers specialize for task‑specific features, and (3) architectural design and training strategies critically affect the ability to retain useful representations for low‑resource languages in deeper layers. They recommend dynamic layer‑wise learning rates, advanced attention variants, and richer multilingual pre‑training data to mitigate degradation observed in models like BLOOM‑1.7B.

Overall, the paper provides a nuanced picture of multilingual model behavior, highlighting both strengths (robust performance on high‑resource languages, effective early‑layer semantics) and weaknesses (deep‑layer degradation, uneven cross‑lingual transfer for typologically distant languages). The findings point toward concrete avenues for improving inclusivity and robustness in future multilingual NLP systems.