小模型也能精准模拟大脑语言反应,压缩后表现依然稳定。
Linguistic properties and model scale in brain encoding: from small to compressed language models
- 用30亿参数小模型即可达到大模型的脑活动预测水平
- 10亿参数以下模型在语义区表现明显下降
- 量化剪枝等压缩技术基本不影响脑响应预测能力
近期研究表明,扩大语言模型规模可提升其与人脑活动的一致性,但驱动这一提升的机制仍不明确。尽管大模型通常表现更好,但其分析难度也大幅增加。本研究系统考察了模型规模和数值精度对脑对齐的影响,比较了全精度大模型、30亿参数小模型(SLMs)及压缩版本(量化、剪枝)在自然语言理解时的fMRI响应预测能力。结果显示,在最大达140亿参数的模型中,30亿参数模型的脑可预测性与更大模型无显著差异,而10亿参数模型在语义相关脑区显著下降。压缩处理中,多数量化和剪枝方法保持神经可预测性,仅有GPTQ例外。语言探针显示,压缩虽损害话语、句法和形态等任务性能,但脑预测性基本不变。总体而言,脑对齐在适度模型规模下即饱和,且对压缩具有强鲁棒性,挑战了关于神经尺度的普遍假设,并为脑对齐语言建模提供了紧凑替代方案。
原文摘要 · Abstract (English)
Recent work has shown that scaling large language models (LLMs) improves their alignment with human brain activity, yet it remains unclear what drives these gains and which representational properties are responsible. Although larger models often yield better task performance and brain alignment, they are increasingly difficult to analyze mechanistically. This raises a fundamental question: what is the minimal model capacity required to capture brain-relevant representations? To address this question, we systematically investigate how constraining model scale and numerical precision affects brain alignment. We compare full-precision LLMs, small language models (SLMs), and compressed variants (quantized and pruned) by predicting fMRI responses during naturalistic language comprehension. Across model families up to 14B parameters, we find that 3B SLMs achieve brain predictivity indistinguishable from larger LLMs, whereas 1B models degrade substantially, particularly in semantic language regions. Brain alignment is remarkably robust to compression: most quantization and pruning methods preserve neural predictivity, with GPTQ as a consistent exception. Linguistic probing reveals a dissociation between task performance and brain predictivity: compression degrades discourse, syntax, and morphology, yet brain predictivity remains largely unchanged. Overall, brain alignment saturates at modest model scales and is resilient to compression, challenging common assumptions about neural scaling and motivating compact models for brain-aligned language modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。