When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners

Multilingual reasoning remains a significant challenge for large language models (LLMs), with performance disproportionately favoring high-resource languages. Drawing inspiration from cognitive neuroscience, which suggests that human reasoning functions largely independently of language processing, we hypothesize that LLMs similarly encode reasoning and language as separable components that can be disentangled to enhance multilingual reasoning. To evaluate this, we perform a causal intervention by ablating language-specific representations at inference time. Experiments on 10 open-weight LLMs spanning 11 typologically diverse languages show that this language-specific ablation consistently boosts multilingual reasoning performance. Layer-wise analyses further confirm that language and reasoning representations can be effectively disentangled throughout the model, yielding improved multilingual reasoning capabilities, while preserving top-layer language features remains essential for maintaining linguistic fidelity. Compared to post-training methods such as supervised fine-tuning or reinforcement learning, our training-free language-reasoning disentanglement achieves comparable or superior results with minimal computational overhead. These findings shed light on the internal mechanisms underlying multilingual reasoning in LLMs and suggest a lightweight and interpretable strategy for improving cross-lingual generalization.

💡 Research Summary

The paper investigates whether large language models (LLMs) encode reasoning and language processing as separable internal components, an idea inspired by cognitive neuroscience findings that human reasoning operates largely independently of language networks. The authors propose a causal intervention that removes language‑specific information from hidden states during inference, thereby disentangling language from reasoning.

To identify language‑specific subspaces, they compute the mean final‑token embedding for each language, concatenate these means into a matrix M, and decompose M into an orthogonal language‑agnostic subspace Mₐ and a language‑specific subspace Mₛ using singular value decomposition (SVD). The dimensionality of Mₛ is chosen to capture rich linguistic variation, while Mₐ is low‑dimensional to reflect shared semantics.

During inference, for any hidden representation h, they apply a projection‑based ablation: ĥ = h − λ Mₛᵀ Mₛ h, where λ controls the strength of removal. This operation is applied layer‑wise without altering model parameters, making it a training‑free, lightweight intervention.

Experiments span ten open‑weight LLMs (including Qwen‑2.5, Qwen‑3, DeepSeek‑R1‑Distill, GLM‑Z1, QwQ‑32B) and eleven languages covering high‑, medium‑, and low‑resource categories (English, Spanish, French, German, Chinese, Japanese, Russian, Thai, Telugu, Bengali, Swahili). The authors evaluate three multilingual reasoning benchmarks: MGSM (mathematical reasoning), XWinograd (commonsense), and M‑MMLU (knowledge‑intensive QA). Accuracy and language fidelity (measured with GlotLID) are reported.

Across almost all model‑language pairs, the language‑reasoning disentanglement yields consistent gains of 1–3 percentage points in accuracy, with especially notable improvements for low‑resource languages. Layer‑wise analysis shows that removing language‑specific components from middle layers provides the best trade‑off: reasoning performance improves while language fidelity remains acceptable. Removing these components from the top layers harms language fidelity, causing the model to default to English outputs, confirming that high‑level layers are crucial for preserving surface language.

The authors compare their method to standard multilingual post‑training approaches such as supervised fine‑tuning and reinforcement learning from human feedback (RLHF). Despite requiring no additional training, the projection‑based intervention matches or exceeds the performance of these heavier methods while incurring negligible computational overhead.

Limitations include the reliance on an English‑biased pretraining corpus, which leads to a tendency for representations to align toward English after ablation; sensitivity to the λ hyperparameter, which may need model‑specific tuning; and the inability to fully preserve language fidelity when top‑layer language signals are removed. Future work is suggested on regularizing subspaces for better cross‑lingual balance, dynamic control of the disentanglement strength, and extending the approach to multimodal settings where reasoning may involve non‑linguistic inputs.

Overall, the study provides compelling empirical evidence that language and reasoning can be functionally separated within LLMs, and that a simple, training‑free projection can serve as an effective tool for enhancing multilingual reasoning capabilities.