通过因果门控解析Transformer注意力头的功能角色。
Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformers
- 基于因果分析为每个注意力头打标签：促进、干扰或无关。
- 在多个任务上验证了头角色的因果性，非简单相关。
- 适合研究模型内部机制与可解释性的研究人员。
我们提出因果头门控（CHG），一种可扩展的方法，用于解析Transformer模型中注意力头的功能角色。CHG通过学习头部的软门控，根据其对任务性能的影响，将头分类为促进、干扰或无关三类。与以往依赖假设、提示模板或目标标签的可解释性方法不同，CHG可直接应用于任意数据集，仅使用标准的下一个词预测任务。我们在Llama 3系列多个大语言模型及多种任务（包括语法、常识推理和数学推理）上评估了CHG，结果表明其得分具有因果意义，经消融实验和因果中介分析验证。我们还引入对比式CHG，以分离特定任务组件的子电路。研究发现：大模型中存在多个稀疏但任务充分的子电路；单个头的角色依赖于与其他头的交互（低模块化）；指令遵循与上下文学习由可分离机制支持。
原文摘要 · Abstract (English)
We present causal head gating (CHG), a scalable method for interpreting the functional roles of attention heads in transformer models. CHG learns soft gates over heads and assigns them a causal taxonomy - facilitating, interfering, or irrelevant - based on their impact on task performance. Unlike prior approaches in mechanistic interpretability, which are hypothesis-driven and require prompt templates or target labels, CHG applies directly to any dataset using standard next-token prediction. We evaluate CHG across multiple large language models (LLMs) in the Llama 3 model family and diverse tasks, including syntax, commonsense, and mathematical reasoning, and show that CHG scores yield causal, not merely correlational, insight validated via ablation and causal mediation analyses. We also introduce contrastive CHG, a variant that isolates sub-circuits for specific task components. Our findings reveal that LLMs contain multiple sparse task-sufficient sub-circuits, that individual head roles depend on interactions with others (low modularity), and that instruction following and in-context learning rely on separable mechanisms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。