提出新型注意力机制，有效去除无关符号噪声，同时保留关键特殊符号信息。
Integral Transformer: Denoising Attention, Not Too Much Not Too Little
- 通过积分采样对数概率分布信号，实现注意力去噪
- 在多个基准上优于原始、Cog及差分注意力方法
- 适合需要稳定注意力分布的语言模型优化场景
Softmax自注意力常为语义无信息的标记（如特殊符号和标点）分配过量权重，称为注意力噪声。现有方法如Cog Attention和差分变压器引入负注意力得分以缓解此问题，但可能丢弃有用信息。本文提出Integral Transformer，一种新型自注意力机制，通过从对数概率分布中采样信号进行积分来去噪。该方法在抑制噪声的同时保留对模型性能至关重要的特殊标记贡献。大量实验表明，该模型在主流知识与推理语言基准上优于原始、Cog及差分注意力变体。分析还发现，在Transformer低层使用原始自注意力可提升性能，且Integral Transformer能有效平衡注意力分布并减少高层的秩坍缩现象。
原文摘要 · Abstract (English)
Softmax self-attention often assigns disproportionate weight to semantically uninformative tokens such as special tokens and punctuation, a phenomenon known as attention noise. While recent methods like Cog Attention and the Differential Transformer have addressed this by introducing negative attention scores, they risk discarding useful information. In this paper, we propose the Integral Transformer, a novel self-attention mechanism that denoises attention by integrating signals sampled from the logit distribution. Our approach mitigates noise while preserving the contributions of special tokens critical for model performance. Extensive experiments demonstrate that our model outperforms vanilla, Cog, and Differential attention variants on well-established knowledge and reasoning language benchmarks. Moreover, our analysis reveals that employing vanilla self-attention in the lower Transformer layers enhances performance and that the Integral Transformer effectively balances attention distributions and reduces rank collapse in upper layers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。