不同规模模型训练时，损失曲线可统一归一化为单一通用曲线。
Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks
- 通过归一化训练算力与损失值，多规模模型损失曲线实现精确收敛。
- 学习率衰减下出现超收敛，跨模型差异低于随机种子噪声水平。
- 适用于评估模型缩放是否最优，尤其适合大模型训练调优。
当模型规模与训练时间同步增长时，神经网络训练动态的缩放极限是什么？我们发现，尽管架构、训练算法与数据之间存在复杂交互，但计算最优训练的模型展现出惊人的普适性。具体而言，将训练算力和损失值在训练结束时归一化为1后，不同规模模型的损失曲线可坍缩为单一通用曲线。引入学习率衰减后，坍缩程度极强，各模型归一化曲线间的差异低于单个模型在不同随机种子下的噪声水平，这一现象称为“超坍缩”。该现象在多种学习率调度、数据集和架构（包括基于下一词预测的Transformer）中均被观测到，并在超参数非最优缩放时失效，提供了一个精确且实用的优质缩放判断指标。我们通过关联坍缩与典型神经网络缩放定律中的幂律结构，结合一个简单却极为有效的随机梯度下降噪声动力学模型，成功预测了多种学习率调度下的损失曲线，并定量解释了超坍缩的起源。
原文摘要 · Abstract (English)
What scaling limits govern neural network training dynamics when model size and training time grow in tandem? We show that despite the complex interactions between architecture, training algorithms, and data, compute-optimally trained models exhibit a remarkably precise universality. Specifically, loss curves from models of varying sizes collapse onto a single universal curve when training compute and loss are normalized to unity at the end of training. With learning rate decay, the collapse becomes so tight that differences in the normalized curves across models fall below the noise floor of individual loss curves across random seeds, a phenomenon we term supercollapse. We observe supercollapse across learning rate schedules, datasets, and architectures, including transformers trained on next-token prediction, and find it breaks down when hyperparameters are scaled suboptimally, providing a precise and practical indicator of good scaling. We explain these phenomena by connecting collapse to the power-law structure in typical neural scaling laws, and analyzing a simple yet surprisingly effective model of SGD noise dynamics that accurately predicts loss curves across various learning rate schedules and quantitatively explains the origin of supercollapse.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。