语言模型在跨语言和跨模态时共享语义空间，实现统一理解。
The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities
- 通过中间层表征发现不同语言的语义等价输入位置相近。
- 共享语义空间可被预训练主导语言解释，且干预一个模态影响其他模态输出。
- 该机制支持多模态统一理解，适合研究跨语言与多模态模型的学者。
现代语言模型能处理多种语言和模态的数据。我们提出语义枢纽假说：模型通过学习异构数据类型（如不同语言、模态）间的共享表示空间，使语义相似的输入即使来自不同模态或语言也彼此靠近。这一结构类比神经科学中的枢纽-辐条模型（Patterson et al., 2007），即人类大脑中语义知识由跨模态的语义‘枢纽’整合各模态特异性区域的信息。我们首先证明，在中间层中，不同语言中语义等价的输入表示相似，并可通过模型主导预训练语言用对数几率透镜进行解释。这种倾向延伸至算术表达式、代码及视觉/音频输入。对某一数据类型的共享表示空间进行干预，可预测地影响其他类型模型输出，表明该共享空间并非大规模训练的副产品，而是模型输入处理过程中主动利用的结构。
原文摘要 · Abstract (English)
Modern language models can process inputs across diverse languages and modalities. We hypothesize that models acquire this capability through learning a shared representation space across heterogeneous data types (e.g., different languages and modalities), which places semantically similar inputs near one another, even if they are from different modalities/languages. We term this the semantic hub hypothesis, following the hub-and-spoke model from neuroscience (Patterson et al., 2007) which posits that semantic knowledge in the human brain is organized through a transmodal semantic "hub" which integrates information from various modality-specific "spokes" regions. We first show that model representations for semantically equivalent inputs in different languages are similar in the intermediate layers, and that this space can be interpreted using the model's dominant pretraining language via the logit lens. This tendency extends to other data types, including arithmetic expressions, code, and visual/audio inputs. Interventions in the shared representation space in one data type also predictably affect model outputs in other data types, suggesting that this shared representations space is not simply a vestigial byproduct of large-scale training on broad data, but something that is actively utilized by the model during input processing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。