用因果干预揭示语言模型对填空句式的共性理解
Causal Interventions Reveal Shared Structure Across English Filler-Gap Constructions
- 通过分布式替换干预，分析语言模型对填空句式的内部机制
- 模型对各类填空句式形成相似的抽象解析，且受频率和上下文影响
- 发现新因素，可推动传统语法理论更新，适合语言学与AI交叉研究者
语言模型（LMs）已成为语言学家构建句法理论的重要证据来源。本文主张，将因果可解释性方法应用于语言模型，可显著提升此类证据的价值，帮助我们刻画模型所学习到的抽象机制。研究聚焦英语填空句式依赖结构（如疑问句、关系从句）。现有语言学理论普遍认为这些结构具有诸多共同特征。基于分布式替换干预的实验表明，语言模型在这些结构上趋于形成相似的抽象分析。这些分析还揭示了以往被忽视的因素——包括词频、填充词类型及周围语境——可能促使标准语言学理论进行调整。总体而言，这一结果表明，对语言模型的机械性内部分析能够推动语言理论的发展。
原文摘要 · Abstract (English)
Language Models (LMs) have emerged as powerful sources of evidence for linguists seeking to develop theories of syntax. In this paper, we argue that causal interpretability methods, applied to LMs, can greatly enhance the value of such evidence by helping us characterize the abstract mechanisms that LMs learn to use. Our empirical focus is a set of English filler-gap dependency constructions (e.g., questions, relative clauses). Linguistic theories largely agree that these constructions share many properties. Using experiments based in Distributed Interchange Interventions, we show that LMs converge on similar abstract analyses of these constructions. These analyses also reveal previously overlooked factors -- relating to frequency, filler type, and surrounding context -- that could motivate changes to standard linguistic theory. Overall, these results suggest that mechanistic, internal analyses of LMs can push linguistic theory forward.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。