提出DNT模型，让Transformer能用普通动量SGD训练且性能不降。
DNT: a Deeply Normalized Transformer that can be trained by Momentum SGD
- 在Transformer中巧妙嵌入归一化，调控梯度分布。
- 在ViT和GPT上验证，可媲美AdamW训练效果。
- 适合想简化训练流程或降低计算开销的研究者。
Transformer已成为现代深度学习的主流骨干网络，但其训练通常依赖自适应学习率优化器（如AdamW），而非普通动量SGDW（mSGDW）。此前研究表明，这主要源于梯度的重尾分布。本文提出深归一化Transformer（DNT），通过在Transformer适当位置集成归一化技术，有效调节各层雅可比矩阵，平衡权重、激活及其相互作用的影响，使梯度分布更加集中。我们提供了所用归一化技术的理论依据，并在两种主流Transformer架构（ViT和GPT）上进行了广泛实验验证：1）DNT性能优于对比模型；2）DNT可被普通mSGDW有效训练，无需复杂优化器。
原文摘要 · Abstract (English)
Transformers have become the de facto backbone of modern deep learning, yet their training typically demands an advanced optimizer with adaptive learning rate like AdamW, rather than a momentum SGDW (mSGDW). Previous works show that it is mainly due to a heavy-tailed distribution of the gradients. In this paper, we introduce a Deeply Normalized Transformer (DNT), which is meticulously engineered to overcome this limitation enabling seamless training with vanilla mSGDW while yielding comparable performance to the Transformers trained via AdamW. To be specific, in DNT, we strategically integrate normalization techniques at proper positions in the Transformers to effectively modulate the Jacobian matrices of each layer, balance the influence of weights, activations, and their interactions, and thus enable the distributions of gradients concentrated. We provide both theoretical justifications of the normalization technique used in our DNT and extensive empirical evaluation on two popular Transformer architectures to validate that: a) DNT outperforms its counterparts (\ie, ViT and GPT), and b) DNT can be effectively trained with vanilla mSGDW.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。