多语言模型用共享电路处理语法一致,跨语言差异越小越相似。
Shared Circuits for Shared Grammar: Tracing Subject-Verb Agreement Across Languages

- 通过注意力头分析,发现语法一致依赖可复用的神经电路。
- 有词形变化的语言共享度更高,英语在需要显式一致时也趋同。
- 相同功能的注意力模式跨语言相似,适合语言学与模型可解释性研究。
多语言大模型常表现出跨语言泛化能力,但其内部机制是否共享仍不明确,尤其在不同语言中语法一致表现差异显著,且英语中该现象较弱。本文针对现在时主谓一致这一形态句法过程,在29种语言和5个开源模型家族中,利用激活修补与注意力分析,识别出参与一致性的因果注意力头,并比较其跨语言特征。结果发现,具有明显人称/数词形变化的语言,其一致性电路更相似;当分析聚焦于词形对比本身时,共享程度最强。英语作为桥梁案例,在需显式一致的情境下变得与变位语言更相似。许多相关注意力头在不同语言中表现出相似模式,表明跨语言重用不仅体现在位置上,还反映在功能角色上。整体表明,多语言大模型并非采用完全独立的语言特解,而是部分复用计算结构来处理形态句法一致。
原文摘要 · Abstract (English)
Multilingual large language models often generalize across languages, and prior work suggests that their internal mechanisms can overlap cross-lingually. It remains unclear, however, when such sharing emerges and whether it varies with the overt realization of the same grammatical operation. We investigate this question for present-tense subject-verb agreement, a morphosyntactic process that varies substantially across languages and is only weakly expressed in English. Using activation patching and attention analysis across 29 languages and five open-source model families, we identify the attention heads causally implicated in agreement and compare these head-level signatures across languages. We find that languages with overt person/number inflection exhibit more similar agreement circuitry than non-conjugating languages, with the strongest sharing appearing when the analysis isolates recovery of the inflectional contrast itself. English provides an informative bridge case, becoming more similar to conjugating languages precisely in contexts where overt agreement is required. Finally, many implicated heads display similar attention patterns across languages, suggesting that cross-lingual overlap reflects shared functional roles as well as shared localization. Together, these results indicate that multilingual LLMs reuse partially shared computational structure for morphosyntactic agreement rather than relying on fully separate language-specific solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。