通过单次前向传播解析Transformer中组件的交互与归属,揭示预测背后的计算路径。
Every Component is a Lookup: Token Attribution and Composition from a Single Decomposition

- 基于注意力与MLP共享的键值结构,反向递归分解组件间交互强度。
- 在Pythia模型中,组件得分可准确预测其被屏蔽后的困惑度提升(斯皮尔曼相关ρ=0.72~0.96)。
- 能区分同名实体在电路中的不同作用,适合研究模型内部机制的学者使用。
Transformer的机械可解释性不仅需识别重要组件,还需揭示它们如何组合成最终预测的计算路径。注意力与MLP均遵循共享的键值模板ϕ(S)U。我们利用此结构提出Unpack方法,通过反向递归从单次前向传播中分解信用,生成任意组件间的交互强度、带K/Q/V标签的端到端路径及每标记的归因,无需干预、梯度或辅助训练。交互得分具有因果基础:在参数量从160M到6.9B的Pythia-deduped系列模型中,组件得分可预测其通信被移除时的困惑度上升(层内斯皮尔曼相关ρ=0.72至0.96)。该方法揭示了Wang等人(2023)提出的间接宾语识别电路的全部三条连接,包括各通道的模式特异性路由:名称移动头处重置路径,S抑制是查询侧最强输入,而在键侧降至第10位,这一差异无法由单一的标记或组件得分表达。将相同方法应用于Hanna等人(2023)的“大于”电路(另一结构),去除第0层写入者后,已知连接仍位列主要贡献者。该分解读出的是实际计算下的贡献,作为因果电路发现的补充而非替代。其每标记归因在输入扰动下保持忠实,性能媲美专用归因方法,并能区分电路机制与表面身份:同一名称出现两次,仅一个驱动电路时获得截然不同的信用。代码已公开于https://github.com/Fun-Cry/unpacklm。
原文摘要 · Abstract (English)
Mechanistic interpretability of transformers requires identifying not just which components matter but how they compose into the computational route that produced a prediction. Both attention and MLP follow a shared key-value template $ϕ(S)U$. We exploit this structure to develop Unpack, a backward recursion that decomposes credit through both sublayers, producing interaction strengths between any two components, named end-to-end paths with K/Q/V composition labels, and per-token attribution, all from a single forward pass, without intervention, gradients, or auxiliary training. The interaction scores are causally grounded: across the Pythia-deduped family from 160M to 6.9B parameters, a component's score predicts the perplexity increase when its communication is ablated (within-layer Spearman $ρ= 0.72$ to $0.96$). The composition paths surface all three connections of the indirect-object-identification circuit of Wang et al. (2023), including the mode-specific routing of each: rerooting at the Name Mover heads, S-Inhibition is the strongest query-side input and falls to rank 10 on the key side, a distinction no single per-token or per-component score can express. The same procedure applied to the greater-than circuit of Hanna et al. (2023), a differently shaped circuit, places the named connections among the top contributors, once layer-0 writers, which carry large credit whatever they feed, are set aside. The decomposition reads out contribution under the realized computation; it complements, rather than performs, causal circuit discovery. The per-token readout is faithful under input perturbation, on par with dedicated attribution methods, and distinguishes circuit mechanism from surface identity: two occurrences of the same name receive radically different credit when only one drives the circuit. Code is available at https://github.com/Fun-Cry/unpacklm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。