语言模型能对跨模态概念一致的大脑区域进行更准确预测。
Language models align with brain regions that represent concepts across modalities
- 通过跨模态一致性衡量大脑区域对同一概念的响应稳定性。
- 语言与图文模型在概念一致性高的脑区表现更优，即使这些区域不敏感于语言。
- 适合研究认知神经科学与大模型内在表征的交叉方向者阅读。
认知科学和神经科学长期面临语言表征与概念意义表征难以分离的问题。当前语言模型（LMs）也面临类似挑战。本文研究了语言模型与大脑活动之间的对齐关系，引入两个神经指标：(1) 句子处理时的大脑激活水平，用于捕捉语言加工；(2) 一种新提出的跨模态意义一致性度量，利用fMRI数据集（Pereira et al., 2018）量化一个脑区在句子、词云、图像等不同输入模态下对同一概念的响应一致性。实验表明，无论是仅语言训练还是语言-视觉联合训练的模型，都能在意义一致性更高的脑区更准确地预测信号，即使这些区域并非强烈响应语言处理。这表明语言模型可能内化了跨模态的概念意义。
原文摘要 · Abstract (English)
Cognitive science and neuroscience have long faced the challenge of disentangling representations of language from representations of conceptual meaning. As the same problem arises in today's language models (LMs), we investigate the relationship between LM--brain alignment and two neural metrics: (1) the level of brain activation during processing of sentences, targeting linguistic processing, and (2) a novel measure of meaning consistency across input modalities, which quantifies how consistently a brain region responds to the same concept across paradigms (sentence, word cloud, image) using an fMRI dataset (Pereira et al., 2018). Our experiments show that both language-only and language-vision models predict the signal better in more meaning-consistent areas of the brain, even when these areas are not strongly sensitive to language processing, suggesting that LMs might internally represent cross-modal conceptual meaning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。