发现语言模型训练中存在损失减速现象，源于梯度相互抵消的零和学习机制。
Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learning
- 提出零和学习（ZSL）机制，解释训练中梯度对立导致损失改善停滞。
- 模型规模越大，损失减速发生越晚，且减速后下降速度更快。
- 该研究对优化模型训练策略有指导意义，尤其适合关注训练动态的开发者。
本文旨在理解语言模型规模扩展如何影响训练动态。我们发现，语言模型在训练早期会出现损失减速现象——即损失改进速率突然放缓，导致损失曲线在双对数空间呈现分段线性特征。增大模型规模可通过（1）降低减速发生的损失值，以及（2）提升减速后的双对数损失下降速率来缓解这一现象。我们将损失减速归因于一种称为零和学习（ZSL）的退化训练动态：每个样本的梯度趋于系统性相反，导致损失在部分样本上的改善会破坏另一部分样本的性能，从而阻碍整体进展。损失减速与零和学习为语言模型缩放规律背后的训练机制提供了新视角，可能独立于规模直接用于改进模型。代码与数据已开源。
原文摘要 · Abstract (English)
This work aims to understand how scaling improves language models, specifically in terms of training dynamics. We find that language models undergo loss deceleration early in training; an abrupt slowdown in the rate of loss improvement, resulting in piecewise linear behaviour of the loss curve in log-log space. Scaling up the model mitigates this transition by (1) decreasing the loss at which deceleration occurs, and (2) improving the log-log rate of loss improvement after deceleration. We attribute loss deceleration to a type of degenerate training dynamics we term zero-sum learning (ZSL). In ZSL, per-example gradients become systematically opposed, leading to destructive interference in per-example changes in loss. As a result, improving loss on one subset of examples degrades it on another, bottlenecking overall progress. Loss deceleration and ZSL provide new insights into the training dynamics underlying language model scaling laws, and could potentially be targeted directly to improve language models independent of scale. We make our code and artefacts available at: https://github.com/mirandrom/zsl
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。