发现模型扩容2倍最高效,超了反而更慢。
When is Warmstarting Effective for Scaling Language Models?

- 用简单方法扩容2倍,比复杂预热更有效。
- 超过2倍扩容后,从头训练反而更快。
- 在每参数20个词以内预算时加速最明显。
从已有检查点扩展模型旨在加速大模型训练,可能节省资源。尽管近年关注度上升,但大规模训练中暖启动(warmstarting)仍应用有限。我们归因于两个未被充分探索的因素:(1) 过分强调保持小模型初始性能,限制了新架构的操作设计;(2) 对扩容与超参数、缩放行为的交互分析不足,且文献中扩容因子不一致。我们证明,保持基线模型的初始性能并非获得优异最终性能所必需,且简单的、与架构无关的扩容策略可优于复杂暖启动算子。关键发现是,存在一个扩容因子 $g$ 的上限,超过该值时从头训练更高效。此现象在多个消融实验中均被观察到,且先前发表结果中已存在但未报告。在密集MLP和语言模型上,$2\times$ 扩容因子最为可靠地实现收敛加速,尤其在每参数20个词以下的预算下效果显著,随预算增加而减弱。我们拟合缩放定律,为从业者提供何时及如何扩容的预测性指导。整体分析提供了实用指南与实证边界。
原文摘要 · Abstract (English)
Model growth from a given checkpoint aims to accelerate training of a larger model, offering potential resource savings. Despite recent interest, warmstarting has seen limited practical adoption in large-scale training. We attribute this to two underexplored factors: (1) an overemphasis on preserving the smaller model's performance at initialization, which constrains operator design for new architectures, and (2) insufficient analysis of how growth interacts with hyperparameters and scaling behavior, compounded by inconsistent growth factors across the literature. We show that preserving the base model's initial post-growth performance is not necessary for strong final performance, and that simple, architecture-agnostic growth strategies can outperform more complex warmstarting operators. Crucially, we empirically identify an upper bound on the growth factor $g$ beyond which training from scratch is more efficient. We observe this across multiple ablation setups. Notably, this limit is also present, but unreported, in prior published results. Across our experiments on dense MLPs and dense language models, we find that a $2\times$ growth factor is the most reliable in yielding convergence speedups, with gains most pronounced under 20 tokens/parameter budgets and diminishing as budget increases. We fit scaling laws over these observations to provide predictive guidance for practitioners deciding when and how much to grow. Together, our analysis provides practical guidelines and empirical limits for model growth.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。