拆解注意力机制的路由与过滤功能,发现其可简化且稳定训练。
The Routing and Filtering Structure of Attention

- 将注意力分解为路由(信息重分配)和过滤(相关性缩放)两部分
- 路由在低秩下运行,首层有效秩仅2,随深度扩展至6个量级
- 新结构支持无归一化训练,可大幅减少参数而保持性能
注意力交互矩阵 $QK^{ op}$ 包含两个纠缠的计算:反对称部分负责位置间信息重分配(路由),对称部分用于相互关联性缩放(过滤)。我们对五种预训练变压器中的1776个注意力头进行分解,发现路由在远低于权重核分配容量的低秩下运行。提出 $S$-$D$ 注意力作为诊断性参数化结构,通过构造实现路由与过滤解耦,保证稳定性($ m{Re}(λ) \ leq 0$),无需层归一化即可稳定训练。解耦后未归一化的路由自发形成谱级联,首层有效秩为2,随深度扩展至六个尺度,参数量从7M增至355M。该级联预测了可简化的区域:线性化前七层125M $S$-$D$ 注意力仅损失<5%困惑度,而标准注意力则崩溃。线性化范围随深度扩大。用ELU+1线性注意力替换前四层,可在全头维度下达到基线的1.4%以内。级联分配架构以47%-65%更少的注意力参数换取+3.9%至+8.4%的困惑度提升。路由-过滤分解使谱预算可见,级联使其可操作。
原文摘要 · Abstract (English)
The attention interaction matrix $QK^{\top}$ contains two entangled computations: a skew-symmetric component that redistributes information between positions (routing) and a symmetric component that scales mutual relevance (filtering). We decompose 1776 heads across five pretrained transformers and find routing operating at low rank, well below the routing capacity allocated by the weight kernel. We introduce $S$-$D$ attention as a diagnostic parameterization that disentangles routing from filtering by construction with guaranteed stability ($\mathrm{Re}(λ) \le 0$) and trains stably without layer normalization. When disentangled and unnormalized, routing self-organizes into a spectral cascade, effective rank $2$ at the first layer, expanding with depth across six scales from 7M to 355M parameters. The cascade predicts where attention can be simplified: linearizing the first seven layers of 125M $S$-$D$ attention costs ${<}5\%$ perplexity, whereas standard attention collapses under the same intervention. The linearizable region widens with depth. Replacing the first four layers with ELU+1 linear attention reaches within $1.4\%$ of baseline at full head dimension. Cascade-allocated architectures trade attention parameters for perplexity ($47\%-65\%$ fewer attention parameters at $+3.9\%$ to $+8.4\%$ PPL). The routing-filtering decomposition makes the spectral budget legible; the cascade makes it actionable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。