arXiv:2603.11487cs.LG2026-03ACL被引 9

softmax注意力机制在特定任务中必然产生注意力汇聚点,这是其数学本质决定的。

Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks

论文配图:Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks
图 1 · 摘自论文原文
  • 通过触发条件任务证明:softmax归一化迫使注意力聚焦于固定位置。
  • 实验显示,softmax模型产生强注意力汇聚,而ReLU注意力无此现象。
  • 适合研究注意力机制原理或模型可解释性的读者参考。

Transformer 模型常表现出注意力汇聚现象:概率质量集中在某个固定且与内容无关的位置。这种现象是优化过程的副产品,还是在某些场景下软最大注意力模型的功能性必需?我们证明,在某些设定下,后者成立:实现简单的触发条件行为必然导致软最大自注意力模型中出现注意力汇聚。该结果形式化了常见直觉:在概率单纯形上的归一化必须将注意力坍缩至一个稳定锚点,以实现默认状态(例如忽略输入)。我们通过一个具体任务验证:当指定触发标记出现时,模型需返回所有先前标记表示的平均值,否则输出零,这一功能与实际注意力头的行为一致(Barbero et al., 2025;Guo et al., 2024)。同时证明,非归一化的ReLU注意力可无需汇聚完成相同任务,确认归一化约束是汇聚行为的根本驱动因素。实验验证了预测,并表明该现象超出理论分析范围:无论单头还是多头,软最大模型均发展出强汇聚,而ReLU注意力则完全消除汇聚。

原文摘要 · Abstract (English)

Transformers often display an attention sink: probability mass concentrates on a fixed, content-agnostic position. Are sinks a byproduct of the optimization/training regime? Or are they sometimes functionally necessary in softmax Transformers? We prove that, in some settings, it is the latter: computing a simple trigger-conditional behavior necessarily induces a sink in softmax self-attention models. Our results formalize a familiar intuition: normalization over a probability simplex must force attention to collapse onto a stable anchor to realize a default state (e.g., when the model needs to ignore the input). We instantiate this with a concrete task: when a designated trigger token appears, the model must return the average of all preceding token representations, and otherwise output zero, a task which mirrors the functionality of attention heads in the wild (Barbero et al., 2025; Guo et al., 2024). We also prove that non-normalized ReLU attention can solve the same task without any sink, confirming that the normalization constraint is the fundamental driver of sink behavior. Experiments validate our predictions and demonstrate they extend beyond the theoretically analyzed setting: softmax models develop strong sinks while ReLU attention eliminates them in both single-head and multi-head variants.

注意力机制Transformer数学分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。