通过因果干预发现多语言模型中语法机制具类型学相似性
Causal Interventions Reveal Typologically Organized Syntactic Mechanisms in Multilingual Language Models
- 用可解释性技术分离语言内语法机制，再跨语言迁移测试
- 三种句法现象在四类模型中均实现跨语言机制迁移
- 迁移程度与语言类型相似度正相关，支持语言共性理论
语言学理论长期认为跨语言存在句法规律，暗示这些结构由相似机制处理。但因缺乏对人类语言处理机制的精细操控手段，这一假说难以实证。本文借助机制可解释性技术，在多语言大模型中研究该问题。首先分离各语言内部的句法机制，再尝试跨语言迁移。针对主谓数一致、代词性别一致和空缺填补三类经典句法现象，在四种模型中均发现稳定的跨语言机制迁移。进一步发现迁移程度呈梯度分布，语言类型越相似，迁移效果越好。本工作为跨语言句法结构和多语言处理提供了新假设，并表明语言模型研究可反哺语言学理论。
原文摘要 · Abstract (English)
Linguistic theory has long recognized cross-linguistic syntactic regularities, leading to claims that these similar structures are processed by similar mechanisms. However, this hypothesis has been difficult to test empirically due to our lack of fine-grained, manipulable access of human processing mechanisms. In this work, we take advantage of techniques from mechanistic interpretability to study such a question in multilingual LMs. We first isolate language-internal mechanisms before attempting to transfer them cross-lingually. Across four models and three well-studied constructions (subject--verb number agreement, anaphoric pronoun gender agreement, and filler--gap object extraction) we find consistent cross-lingual mechanism transfer. We further find transfer to be graded, with more transfer between more typologically similar languages. We believe our work provides novel hypotheses about cross-linguistic syntactic structures and multilingual processing, and more broadly shows how the study of language models can help inform linguistic theory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。