通过近似归一化提升Transformer训练速度，无需调参且收敛快40%。
Learning in Compact Spaces with Approximately Normalized Transformer
- 用简单缩放实现近似归一化，避免严格约束参数
- 相比GPT模型提速40%，仅增加3%运行时间
- 支持更大批次训练，适合大规模模型开发
深度神经网络训练常面临过拟合、数值不稳定性及残差流方差增大等问题。传统方法依赖正则化与归一化，需调整额外超参数。本文提出一种基于高维随机向量范数集中特性的近似归一化方法，通过简单的标量乘法实现，无需严格约束参数。该方法省去权重衰减和学习率预热，同时不增加归一化层数量。在Transformer架构上的实验表明，相比采用QK归一化的GPT模型，收敛速度提升最高达40%，额外运行开销仅为3%。在推导缩放定律时发现，该方法可在保持经典GPT架构优良缩放特性的同时支持更大批量训练。
原文摘要 · Abstract (English)
The successful training of deep neural networks requires addressing challenges such as overfitting, numerical instabilities leading to divergence, and increasing variance in the residual stream. A common solution is to apply regularization and normalization techniques that usually require tuning additional hyperparameters. An alternative is to force all parameters and representations to lie on a hypersphere. This removes the need for regularization and increases convergence speed, but comes with additional costs. In this work, we propose a more holistic, approximate normalization via simple scalar multiplications motivated by the tight concentration of the norms of high-dimensional random vectors. Additionally, instead of applying strict normalization for the parameters, we constrain their norms. These modifications remove the need for weight decay and learning rate warm-up as well, but do not increase the total number of normalization layers. Our experiments with transformer architectures show up to 40% faster convergence compared to GPT models with QK normalization, with only 3% additional runtime cost. When deriving scaling laws, we found that our method enables training with larger batch sizes while preserving the favorable scaling characteristics of classic GPT architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。