用权重直接绘制注意力模型的通信图谱,揭示关键信息通道。
The Communication Map of a Transformer

- 从权重推导出所有潜在通信路径,统一量化18类连接。
- 发现70%-89%头对方向非随机,强耦合或主动回避。
- 可定位诱导回路并干预,验证其对上下文学习的关键作用。
Transformer 通过共享残差流进行组件间通信,传统可解释性研究需人工逐个分析电路。本文提出通信图谱,仅凭模型权重即可刻画语言模型中所有潜在通信通道,将此前的组合评分扩展为涵盖18种连接类型的单一耦合系数。对GPT-2(约6.3×10⁸条)和Pythia-6.9B(约1.3×10¹¹条)的候选通道普查显示,70%-89%的头对方向显著偏离随机,部分强耦合,部分则主动避开。全图生成耗时15秒(GPT-2)和11分钟(Pythia-6.9B),可在单张消费级显卡上完成。两个应用验证其价值:应用一中最强头-头耦合可无监督恢复已知诱导电路,并将其聚类为社区,删除该社区会破坏模型的上下文复制能力;应用二中,聚合所有头的耦合系数识别出一个独特二维流子空间,删除该空间会使六款模型(最高至Pythia-6.9B)丧失诱导能力,且此空间不同于激活PCA或异常维度所识别的结果。代码、统计工具与干预套件均已开源。
原文摘要 · Abstract (English)
The components of a transformer communicate by writing to and reading from a shared residual stream, and mechanistic interpretability has mapped these connections by hand, one circuit at a time. We present the communication map, which charts every potential communication channel in a language model from weights alone, generalizing the composition score of Elhage et al. (2021) into a single coupling coefficient covering all 18 connection classes, from entire attention head circuits to single neurons. The census of all candidate channels, from $6.3\times10^{8}$ in GPT-2 to $1.3\times10^{11}$ in Pythia-6.9B, finds that 70-89% of head pairs are oriented far from chance, some coupled strongly and others actively avoiding each other. The full map costs 15 seconds for GPT-2 and 11 minutes for Pythia-6.9B on one consumer GPU. Two applications demonstrate the utility of the map. In Application 1, the strongest head-to-head couplings recover the known induction circuits blind and group them into communities, and ablating one such community destroys the model's in-context copying. In Application 2, pooling every head's coupling coefficients identifies a distinct two-dimensional stream subspace, whose deletion abolishes the induction capability in six models up to Pythia-6.9B. This subspace is different from those identified by either activation PCA or outlier dimensions. We release the map, the statistical machinery, and the intervention suite.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。