提出QUEST注意力机制，让模型训练更稳定且更抗干扰。
QUEST: A robust attention formulation using query-modulated spherical attention
- 用球面空间约束键向量，查询动态调节注意力锐度
- 训练无发散，性能提升，对噪声和对抗攻击更鲁棒
- 可直接替换标准注意力，适合视觉等多领域应用
Transformer 模型广泛应用于深度学习，其核心是注意力机制。标准注意力通过缩放点积后使用 softmax 进行计算，但查询和键向量的范数任意增长可能导致训练不稳定。我们证明即使在简单 Transformer 中，数据中易学的伪模式也会引发此类问题。为此提出新型注意力机制——查询调制球面注意力（QUEST），将键向量限制在超球面潜在空间，同时允许每个标记灵活控制注意力分布的尖锐程度。QUEST 可作为标准注意力的即插即用替代方案。我们在视觉任务中重点验证，也拓展至其他领域以展示其通用性。实验表明：（1）QUEST 训练过程无不稳定现象；（2）模型性能显著提升；（3）对数据扰动和对抗攻击具有更强鲁棒性。
原文摘要 · Abstract (English)
The Transformer model architecture has become one of the most widely used in deep learning and the attention mechanism is at its core. The standard attention formulation uses a softmax operation applied to a scaled dot product between query and key vectors. We explore the role played by norms of the queries and keys, which can cause training instabilities when they arbitrarily increase. We demonstrate how this can happen even in simple Transformer models, in the presence of easy-to-learn spurious patterns in the data. We propose a new attention formulation, QUEry-modulated Spherical aTtention (QUEST), that constrains the keys to a hyperspherical latent space, while still allowing individual tokens to flexibly control the sharpness of the attention distribution. QUEST can be easily used as a drop-in replacement for standard attention. We focus on vision applications while also exploring other domains to highlight the method's generality. We show that (1) QUEST trains without instabilities and (2) produces models with improved performance (3) that are robust to data corruptions and adversarial attacks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。