发现大模型训练中损失曲线可归一化为统一轨迹，预示高效训练。
Scaling with Collapse: Efficient and Predictable Training of LLM Families
- 通过归一化损失曲线，实现跨规模训练路径的精确对齐。
- 最优超参下曲线塌缩，与计算效率直接相关。
- 可用于早期诊断训练问题和加速超参调优，适合大规模模型研发者。
有效的大语言模型训练依赖于关键量（如最终损失、最优超参数）随模型和数据规模的可预测缩放。Qiu 等人（2025）最近表明，这种可预测性可超越标量：经过简单归一化后，整个训练损失曲线可“塌缩”到一条通用轨迹上。然而，尚不清楚该现象是否在实际缩放方案中持续存在，即宽度、深度、学习率、批量大小和权重衰减联合缩放时。我们证明它确实存在：当优化超参数针对给定数据预算设置为最优时，损失曲线在不同规模间精确塌缩，这与近期的经验缩放定律一致。因此，塌缩成为计算效率训练的标志。我们在大规模上展示了两个应用：(1) 塌缩偏差可作为训练异常的敏感早期诊断工具；(2) 塌缩曲线的可预测性支持大规模超参调优中的早期停止。最后，我们基于这些洞察训练出一个性能优异的 LLM 家族 *Celerity*，确立了塌缩作为高效开发大模型的有效工具。
原文摘要 · Abstract (English)
Effective LLM training depends on predictable scaling of key quantities -- such as final loss and optimal hyperparameters -- with model and dataset size. Qiu et al. (2025) recently showed that this predictability can extend beyond scalars: whole training loss curves can *collapse* onto a universal trajectory after a simple normalization. What remains unclear is whether this phenomenon persists for LLM families trained under *practical scaling recipes*, where width, depth, learning rate, batch size, and weight decay are scaled jointly. We show that it does: loss curves collapse across scales precisely when optimization hyperparameters are set optimally for the given data budget, in accordance with recent empirical scaling laws. Collapse therefore emerges as a signature of compute-efficient training. We demonstrate two applications at scale: (1) deviation-from-collapse provides a sensitive, early diagnostic of training pathologies, and (2) predictability of collapsed curves enables early stopping in large-scale hyperparameter tuning. Finally, we train a competitive LLM family, *Celerity*, using these insights, establishing collapse as an effective tool for developing efficient LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。