用多项式替代Softmax，让Transformer更稳定训练
Rethinking Attention: Polynomial Alternatives to Softmax in Transformers
- 用多项式激活实现类似Softmax的矩阵范数正则化
- 多项式注意力在多种任务上表现接近甚至超越Softmax
- 适合研究注意力机制原理或追求模型稳定性的开发者
本文质疑Softmax注意力在Transformer中表现优异是否源于其生成输入的概率分布。我们提出，Softmax的有效性实际来自其对注意力矩阵Frobenius范数的隐式正则化，这有助于训练稳定。受此启发，我们探索了多项式类激活函数，发现某些多项式能实现相似正则化效果。理论分析表明，这些多项式虽不满足Softmax的正性、归一化和稀疏性，但在多个Transformer应用中仍表现出色。大量实验验证了该观点，为注意力机制提供了新视角。
原文摘要 · Abstract (English)
This paper questions whether the strong performance of softmax attention in transformers stems from producing a probability distribution over inputs. Instead, we argue that softmax's effectiveness lies in its implicit regularization of the Frobenius norm of the attention matrix, which stabilizes training. Motivated by this, we explore alternative activations, specifically polynomials, that achieve a similar regularization effect. Our theoretical analysis shows that certain polynomials can serve as effective substitutes for softmax, achieving strong performance across transformer applications despite violating softmax's typical properties of positivity, normalization, and sparsity. Extensive experiments support these findings, offering a new perspective on attention mechanisms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。