通过参数重参数化缓解大模型预训练中的损失突增问题
Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes
- 引入门控参数实现权重缩放，使参数范数均匀化
- 在1.3亿至130亿参数模型上训练更稳定且加速收敛
- 适合关注大模型训练稳定性与初始化方法的研究者
损失突增是大语言模型预训练中的核心问题。本文认为参数范数不均匀是导致该现象的原因之一。在神经网络训练中，需保持各层梯度尺度恒定以避免梯度消失或爆炸。然而，Transformer 模型为满足此要求，需采用非均匀参数范数，导致小范数参数对更新更敏感。为此，我们提出一种新方法——权重缩放重参数化（WeSaR），为每个参数矩阵引入门控参数并调节至满足条件的值。由于门控机制，WeSaR 使原始参数范数统一，从而实现稳定训练。在包含1300万、13亿和130亿参数的Transformer解码器上的实验表明，WeSaR 能有效稳定并加速训练，性能优于包括主流初始化方法在内的现有方法。
原文摘要 · Abstract (English)
Loss spikes, a phenomenon in which the loss value diverges suddenly, is a fundamental issue in the pre-training of large language models. This paper supposes that the non-uniformity of the norm of the parameters is one of the causes of loss spikes. Here, in training of neural networks, the scale of the gradients is required to be kept constant throughout the layers to avoid the vanishing and exploding gradients problem. However, to meet these requirements in the Transformer model, the norm of the model parameters must be non-uniform, and thus, parameters whose norm is smaller are more sensitive to the parameter update. To address this issue, we propose a novel technique, weight scaling as reparameterization (WeSaR). WeSaR introduces a gate parameter per parameter matrix and adjusts it to the value satisfying the requirements. Because of the gate parameter, WeSaR sets the norm of the original parameters uniformly, which results in stable training. Experimental results with the Transformer decoders consisting of 130 million, 1.3 billion, and 13 billion parameters showed that WeSaR stabilizes and accelerates training and that it outperformed compared methods including popular initialization methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。