通过调控注意力层的谱特性，提升Transformer的性能表现。
Spectral Conditioning of Attention Improves Transformer Performance
- 基于注意力块雅可比矩阵的理论分析，提出谱调节方法。
- 降低雅可比条件数，显著改善模型训练稳定性与收敛速度。
- 无需修改结构，适配各类注意力机制和任务场景。
我们对Transformer中注意力模块的雅可比矩阵进行了理论分析，发现其受查询、键、值投影的支配。基于此洞察，我们提出一种系统性调整每层注意力谱特性的方法，以降低雅可比条件数，从而改善Transformer内注意力层的整体条件性。实证表明，这种改进的雅可比条件性可转化为实际性能提升。该方法简单通用，可作为现有注意力机制的即插即用替换。我们在多种Transformer架构与任务上验证了其有效性，均实现稳定性能增益。
原文摘要 · Abstract (English)
We present a theoretical analysis of the Jacobian of an attention block within a transformer, showing that it is governed by the query, key, and value projections that define the attention mechanism. Leveraging this insight, we introduce a method that systematically alters the spectral properties of each attention layer to reduce the Jacobian's condition number, thereby improving the overall conditioning of the attention layers within a transformer network. We empirically show that this improved Jacobian conditioning translates to enhanced performance in practice. Our approach is simple, broadly applicable, and can be easily integrated as a drop-in replacement for a wide range of existing attention mechanisms. We validate its effectiveness across diverse transformer architectures and tasks, demonstrating consistent improvements in performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。