用正交扩展法回收旧模型，提升大模型预训练效率
Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
- 通过复制层和噪声专家扩容，正交扩展已训练模型
- 700亿参数模型在1万亿词上比从头训练高10.6%准确率
- 适合已有模型资产但算力有限的研究者
随着大语言模型（LLM）预训练计算需求持续增长，高效训练范式愈发关键。尽管已有大量资源投入预训练检查点，但受架构限制，这些资产常被低估。本文提出一种“正交扩展”策略，通过有策略地扩展参数来“回收”现有检查点，以支持后续训练。方法聚焦于优化收敛的混合专家（MoE）模型，从两个维度入手：跨层复制以增加深度，噪声专家复制以扩大宽度。通过广泛的缩放律分析，我们发现前期投资（沉没成本）与最终模型准确率存在强正相关。在高达700亿参数、1万亿词的数据集上进行实证，该回收方法在相同额外算力预算下，相比从头训练提升10.6%准确率。本工作为可持续的大规模LLM开发提供了低成本蓝图。
原文摘要 · Abstract (English)
As the computational demands for pre-training Large Language Models (LLMs) continue to surge, the need for efficient training paradigms becomes critical. Despite the vast resources already invested in existing pre-trained checkpoints, these assets often remain under-leveraged due to architectural limitations. We introduce an "orthogonal growth" strategy designed to "recycle" these checkpoints by strategically expanding their parameters prior to continued training. Our method focuses on optimizing converged Mixture-of-Experts (MoE) models through two dimensions: interpositional layer copying for increased depth and noisy expert duplication for expanded width. Through extensive scaling laws analysis, we demonstrate a strong positive correlation between the "sunk cost" (prior investment) and the final model accuracy. Empirical results on models up to 70B parameters and 1T tokens show that our recycling approach yields a 10.6% accuracy improvement compared to training from scratch under identical extra compute budgets. This work provides a cost-effective blueprint for sustainable large-scale LLM development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。