Linguistic properties and model scale in brain encoding: from small to compressed language models

Recent work has shown that scaling large language models (LLMs) improves their alignment with human brain activity, yet it remains unclear what drives these gains and which representational properties are responsible. Although larger models often yield better task performance and brain alignment, they are increasingly difficult to analyze mechanistically. This raises a fundamental question: what is the minimal model capacity required to capture brain-relevant representations? To address this question, we systematically investigate how constraining model scale and numerical precision affects brain alignment. We compare full-precision LLMs, small language models (SLMs), and compressed variants (quantized and pruned) by predicting fMRI responses during naturalistic language comprehension. Across model families up to 14B parameters, we find that 3B SLMs achieve brain predictivity indistinguishable from larger LLMs, whereas 1B models degrade substantially, particularly in semantic language regions. Brain alignment is remarkably robust to compression: most quantization and pruning methods preserve neural predictivity, with GPTQ as a consistent exception. Linguistic probing reveals a dissociation between task performance and brain predictivity: compression degrades discourse, syntax, and morphology, yet brain predictivity remains largely unchanged. Overall, brain alignment saturates at modest model scales and is resilient to compression, challenging common assumptions about neural scaling and motivating compact models for brain-aligned language modeling.

💡 Research Summary

This paper investigates how model size and numerical precision affect the alignment between language models and human brain activity measured with fMRI during naturalistic story listening. The authors ask three research questions: (1) What is the minimal model capacity required to achieve brain‑model alignment comparable to large language models (LLMs)? (2) How do compression techniques—post‑training quantization and unstructured pruning—impact brain alignment for both small language models (SLMs) and LLMs? (3) Which linguistic properties are preserved or degraded across model scales and compression, and do these changes correlate with brain alignment?

To answer these questions, the study uses a publicly available fMRI dataset (Deniz et al., 2019) in which nine participants listened to narratives from the Moth Radio Hour. The data are analyzed with the Glasser Atlas, focusing on 180 regions of interest (ROIs), including eight language‑related areas (angular gyrus, inferior frontal gyrus, lateral temporal cortex, etc.). Voxel‑wise encoding models are trained using bootstrap ridge regression to predict fMRI responses from model representations, and decoding experiments reconstruct linguistic representations and text from brain activity.

Three modern transformer families—LLaMA‑3.2, Qwen‑2.5, and DeepSeek‑R1—are evaluated across six scales ranging from 1 B to 14 B parameters. For each family, the authors include small language models (1 B–3 B) and larger counterparts (7 B–14 B). All models are base (non‑instruction‑tuned) checkpoints. Representations are extracted from every layer, and the single layer that yields the highest brain predictivity is selected for each model, ensuring fair cross‑architecture comparison.

Compression is applied post‑training without any fine‑tuning. Three quantization methods are examined: Activation‑aware Weight Quantization (AWQ), GPTQ, and SmoothQuant. Additionally, magnitude‑based unstructured pruning is performed at sparsity levels of 10 %, 25 %, and 50 %. Model sizes after compression are reported in gigabytes.

Linguistic competence is measured with FlashHolmes, a streamlined benchmark covering 66 phenomena across five categories: morphology, syntax, semantics, discourse, and reasoning. This provides a task‑level view of how compression affects language abilities.

Key Findings

1. Early Saturation of Brain Alignment
   Across all three families, 3 B‑parameter SLMs achieve brain‑model alignment indistinguishable from 7 B–14 B LLMs. Normalized Pearson correlations (brain prediction divided by the cross‑subject ceiling) are around 0.45–0.48 for 3 B models and do not differ significantly from larger models in any ROI. In contrast, 1 B models show a marked drop (≈0.32) especially in semantic regions such as the angular gyrus and lateral temporal cortex. Decoding results mirror this pattern: 3 B models can reconstruct semantically coherent text from fMRI, while 1 B models produce fragmented, less meaningful output.

2. Robustness to Compression, with One Exception
   Most quantization schemes (AWQ, SmoothQuant) and pruning up to 25 % sparsity preserve brain predictivity, with changes typically within ±0.02 of the full‑precision baseline. Pruning at 50 % sparsity leads to a noticeable decline, especially for the smallest models. GPTQ, however, consistently reduces brain alignment across all scales, causing the largest drops in semantic ROIs (e.g., angular gyrus). This suggests that GPTQ’s gradient‑guided weight rounding disrupts representations critical for neural prediction.

3. Dissociation Between Linguistic Performance and Brain Alignment
   Compression degrades performance on FlashHolmes tasks, particularly in discourse, syntax, and morphology (average drops of 4–7 %). Nevertheless, these degradations do not translate into reduced brain predictivity. Even models that perform worse on linguistic benchmarks retain similar fMRI encoding scores. This indicates that the neural representations driving brain alignment are not tightly coupled to the surface linguistic competencies captured by standard probing tasks; instead, they may reflect higher‑level semantic integration that is more resilient to parameter reduction.

4. Consistency Across Model Families
   The early saturation at ~3 B parameters and the compression patterns hold for LLaMA, Qwen, and DeepSeek, suggesting that the findings are architecture‑agnostic rather than specific to a single model design.

Implications

• Efficient Brain‑Aligned Modeling: Researchers can use compact (~3 B) models without sacrificing neural predictivity, dramatically lowering computational and memory requirements for neuro‑AI studies.
• Compression Strategy Guidance: AWQ and SmoothQuant are safe choices for preserving brain alignment, while GPTQ should be avoided when the goal is neural fidelity.
• Rethinking Evaluation Metrics: Standard linguistic probing may not be sufficient to gauge a model’s suitability for brain‑model alignment; dedicated neural metrics are necessary.
• Future Directions: Extending the framework to multimodal stimuli, exploring fine‑tuning on neural data, and investigating the specific representational dimensions that survive compression could further illuminate the relationship between artificial and biological language processing.

In summary, the study demonstrates that brain‑language model alignment saturates at modest model scales, remains robust to most compression techniques, and can diverge from conventional linguistic performance measures. These insights pave the way for more resource‑efficient and theoretically grounded models in cognitive neuroscience and AI.