The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?

Multilingual language models (LMs) promise broader NLP access, yet current systems deliver uneven performance across the world’s languages. This survey examines why these gaps persist and whether they reflect intrinsic linguistic difficulty or modeling artifacts. We organize the literature around two questions: do linguistic disparities arise from representation and allocation choices (e.g., tokenization, encoding, data exposure, parameter sharing) rather than inherent complexity; and which design choices mitigate inequities across typologically diverse languages. We review linguistic features, such as orthography, morphology, lexical diversity, syntax, information density, and typological distance, linking each to concrete modeling mechanisms. Gaps often shrink when segmentation, encoding, and data exposure are normalized, suggesting much apparent difficulty stems from current modeling choices. We synthesize these insights into design recommendations for tokenization, sampling, architectures, and evaluation to support more balanced multilingual LMs.

💡 Research Summary

The paper provides a comprehensive survey of why multilingual language models (LLMs) exhibit systematic performance gaps across languages and argues that most of these gaps stem from design choices rather than intrinsic linguistic difficulty. The authors structure the discussion around two central questions: (1) Are observed disparities primarily due to the inherent complexity of certain languages (e.g., rich morphology, complex syntax), or are they artifacts of how models are built and trained (tokenization, encoding, data allocation, parameter sharing)? (2) Which architectural or training decisions can reduce these inequities?

First, the survey reviews key linguistic properties—orthography, morphology, lexical diversity, syntax, information density, and typological distance—and links each to concrete modeling mechanisms. Orthography influences encoding efficiency: Latin scripts occupy a single UTF‑8 byte, whereas scripts such as Chinese, Devanagari, or Arabic require two to three bytes. Under a fixed token budget, this “byte premium” reduces effective exposure for non‑Latin languages, inflates sequence length, and shrinks the effective context window. Moreover, sub‑word tokenizers (BPE, WordPiece) that operate on byte‑level merges favor high‑resource Latin scripts, leading to larger, more informative sub‑words for those languages while fragmenting words in byte‑heavy scripts. The authors cite recent work (e.g., Script‑BPE, MYTE) that mitigates these penalties by constructing script‑aware vocabularies or normalizing byte length during sampling.

Morphological richness is another major factor, but the paper shows that its impact is mediated by tokenization quality. Languages with extensive inflection, derivation, or compounding produce many surface forms, which under frequency‑based sub‑word tokenizers become sparsely represented, raising perplexity. However, when morphology‑aware segmenters (Morfessor, morphological BPE) are employed, the surprisal gap largely disappears. Experiments demonstrate that controlling for tokenization, encoding, and effective exposure eliminates most of the performance difference between morphologically simple and complex languages, suggesting that morphology per se is not a fundamental barrier.

Lexical diversity, measured via type‑token ratios or vocabulary size, also predicts difficulty, but again the effect is largely an artifact of how sub‑word tokenizers split compounds and multi‑word expressions. Languages that lexicalize concepts as long compounds (e.g., German, Finnish) suffer inflated token counts under naive BPE, whereas character‑level or byte‑level models reduce this disparity.

Syntactic variation influences performance indirectly. When languages differ strongly in word order, case marking, or dependency structures, shared‑parameter training can cause negative transfer: gradients from one language interfere with another, leading to representation collapse. The survey highlights modular solutions such as language adapters, typology‑aware routing, and controlled parameter sharing to alleviate this issue.

Data sampling and evaluation practices further exacerbate gaps. High‑resource languages dominate the training corpus, giving them disproportionate exposure. Additionally, standard perplexity metrics conflate tokenization and encoding differences, making cross‑lingual comparisons unfair. The authors recommend token‑level normalization of training data, script‑balanced vocabulary construction, and the inclusion of character‑ or morpheme‑level evaluation metrics alongside traditional token‑based scores.

Synthesizing these findings, the paper proposes concrete design recommendations: (1) adopt script‑balanced or hybrid tokenizers that respect morphological boundaries; (2) perform byte‑normalized or information‑normalized sampling to ensure equal effective exposure across languages; (3) incorporate modular capacity (language adapters, typology‑aware routing) to prevent negative transfer in highly divergent language pairs; (4) revise evaluation protocols to report character‑ and morpheme‑level perplexities and include tokenization diagnostics.

Overall, the survey concludes that multilingual LLM performance disparities are rarely a manifestation of intrinsic linguistic hardness. Instead, they arise from three intertwined mechanisms: (i) inefficient encoding and tokenization that penalize certain scripts; (ii) shared‑parameter training that cannot accommodate extreme typological diversity without dedicated capacity; and (iii) data sampling and evaluation practices that misrepresent true language exposure. By redesigning tokenization, sampling, architecture, and evaluation with typological awareness, future multilingual models can achieve far more balanced performance across the world’s languages.