揭示注意力机制中sink与对角模式如何防止过平滑
Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention
- 通过几何分析发现sink需与所有嵌入对齐才能实现
- 证明密集注意力比稀疏注意力更易导致过平滑，且实际中常见
- 阐明sink等价于硬注意力切换，且自通信可缓解其限制
本文研究sink和对角模式在注意力切换与防过平滑中的作用。分析了sink存在的几何条件，表明sink嵌入必须与所有其他嵌入对齐。进一步细化了sink防过平滑的机制：证明密集注意力在特定条件下必然比稀疏注意力更易导致过平滑，并实证验证这些条件在实践中常成立。还证明了sink与硬注意力切换等价，即注意力输出恒为0。最后放松硬切换约束，允许令牌自通信，定量比较了表示sink与对角模式的成本，解释为何预训练Transformer更偏好sink。引入对角模式分析并推广注意力切换机制，弥合了过平滑预防需求与sink功能之间的差距，同时明确了在无需令牌通信时，注意力层为何会表现如MLP。
原文摘要 · Abstract (English)
This paper studies the role of sinks and diagonal patterns as attention switch and anti-oversmoothing mechanisms. We analyze geometric conditions under which sinks can be represented, showing a necessary alignment between the embedding of the sink and all other embeddings. Next, we refine the current understanding of the role of sinks in oversmoothing prevention: we specify the conditions under which dense attention provably smooths more than sparse attention, and empirically verify that such conditions are often satisfied in practice. We further prove an equivalence between sinks and hard attention switch, in which the output of the attention is identically 0. Finally, we relax the hard attention switch by allowing token self-communication: we provide a quantitative comparison of the costs of representing sinks vs.\ diagonal patterns, showing why sinks are favored in pretrained transformers. The introduction and analysis of diagonal patterns and the generalization of the attention switch close the gap between what oversmoothing prevention requires and what sinks provide, while also establishing when and why attention layers act like MLPs if token communication is not necessary.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。