通过优化嵌入令牌改善注意力机制，提升Transformer训练稳定性与效率
Enhancing Transformers Through Conditioned Embedded Tokens
- 提出条件化嵌入令牌方法，从数据层面改善注意力块的数值条件
- 在图像分类、目标检测等任务上实现训练更稳定、收敛更快
- 适用于各类Transformer架构，尤其适合追求高效训练的研究者
Transformer已深刻改变现代机器学习，在计算机视觉、自然语言处理和机器人等领域取得突破。其成功核心在于注意力机制，可建模输入标记间的全局依赖。然而我们发现，Transformer中的注意力模块存在固有的病态条件问题，阻碍基于梯度的优化，导致训练效率低下。为此，我们建立了一个理论框架，揭示注意力模块的条件性与嵌入标记数据条件性的直接关联。基于此，我们提出条件化嵌入令牌方法，系统性地调整嵌入标记以改善注意力机制的条件性。分析表明，该方法显著缓解了病态条件，带来更稳定高效的训练。我们在多种Transformer架构上验证了该方法，在图像分类、目标检测、实例分割及自然语言处理任务中均取得一致改进，证明其广泛适用性和有效性。
原文摘要 · Abstract (English)
Transformers have transformed modern machine learning, driving breakthroughs in computer vision, natural language processing, and robotics. At the core of their success lies the attention mechanism, which enables the modeling of global dependencies among input tokens. However, we reveal that the attention block in transformers suffers from inherent ill-conditioning, which hampers gradient-based optimization and leads to inefficient training. To address this, we develop a theoretical framework that establishes a direct relationship between the conditioning of the attention block and that of the embedded tokenized data. Building on this insight, we introduce conditioned embedded tokens, a method that systematically modifies the embedded tokens to improve the conditioning of the attention mechanism. Our analysis demonstrates that this approach significantly mitigates ill-conditioning, leading to more stable and efficient training. We validate our methodology across various transformer architectures, achieving consistent improvements in image classification, object detection, instance segmentation, and natural language processing, highlighting its broad applicability and effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。