通过引入预条件矩阵，提升Transformer注意力机制的数值稳定性。
Preconditioned Attention: Enhancing Efficiency in Transformers
- 在每个注意力头中加入条件矩阵，改善矩阵病态性。
- 理论证明可显著降低注意力矩阵的条件数。
- 兼容多种注意力机制，适合高效训练场景。
Transformer成功的关键在于注意力模块，它能有效建模输入词元间的全局依赖关系。然而，我们从理论上证明，标准注意力机制常产生条件数过大的病态矩阵，这会阻碍基于梯度的优化器，导致训练效率低下。为此，我们提出预条件注意力（preconditioned attention），在每个注意力头中引入一个条件矩阵。理论分析表明，该方法能显著降低注意力矩阵的条件数，获得更良态的矩阵，从而提升优化性能。预条件注意力可作为文献中多种注意力机制的简单替换方案。我们在图像分类、目标检测、实例分割、长序列建模和语言建模等多样化的Transformer应用中验证了其有效性。
原文摘要 · Abstract (English)
Central to the success of Transformers is the attention block, which effectively models global dependencies among input tokens associated to a dataset. However, we theoretically demonstrate that standard attention mechanisms in transformers often produce ill-conditioned matrices with large condition numbers. This ill-conditioning is a well-known obstacle for gradient-based optimizers, leading to inefficient training. To address this issue, we introduce preconditioned attention, a novel approach that incorporates a conditioning matrix into each attention head. Our theoretical analysis shows that this method significantly reduces the condition number of attention matrices, resulting in better-conditioned matrices that improve optimization. Conditioned attention serves as a simple drop-in replacement for a wide variety of attention mechanisms in the literature. We validate the effectiveness of preconditioned attention across a diverse set of transformer applications, including image classification, object detection, instance segmentation, long sequence modeling and language modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。