用共识机制替代注意力，让Transformer训练更稳定。
Stabilizing Transformer Training Through Consensus
- 用共识机制替代注意力，提升训练稳定性
- 在文本、DNA、蛋白质数据上均实现更宽学习率范围的稳定训练
- 适合追求训练鲁棒性的模型开发者使用
标准注意力机制的Transformer在学习率过高的情况下容易出现训练不稳，尽管已有方法通过优化过程改进抗过调能力，但架构层面的创新仍较少。本文提出以共识机制作为注意力的即插即用替代方案，能显著提升Transformer在更广泛学习率范围内的训练稳定性。我们将共识机制建模为图模型，并通过大量实验验证其在文本、DNA和蛋白质模态上的有效性。进一步提出了混合共识-注意力框架，在保持性能的同时增强稳定性。还提供了理论分析，揭示了共识机制的内在特性。
原文摘要 · Abstract (English)
Standard attention-based transformers are known to exhibit instability under learning rate overspecification during training, particularly at high learning rates. While various methods have been proposed to improve resilience to such overspecification by modifying the optimization procedure, fundamental architectural innovations to this end remain underexplored. In this work, we illustrate that the consensus mechanism, a drop-in replacement for attention, stabilizes transformer training across a wider effective range of learning rates. We formulate consensus as a graphical model and provide extensive empirical analysis demonstrating improved stability across learning rate sweeps on text, DNA, and protein modalities. We further propose a hybrid consensus-attention framework that preserves performance while improving stability. We provide theoretical analysis characterizing the properties of consensus.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。