研究大模型复用小模型训练的效率极限，发现数据量与配置间的相互作用会制约性能提升。
Scaling Laws for Upcycling Mixture-of-Experts Language Models
- 通过实验发现大模型复用小模型时，数据量与配置存在新交互项
- 在大规模计算预算下，复用效率受数据组合影响而下降
- 给出复用优于从零训练的条件，指导资源有限场景下的模型扩展
预训练大型语言模型（LLM）耗时耗力，常需高配GPU集群数月。现有两种减负路径：复用小模型训练大模型（即“向上升级”），或训练高效模型如混合专家（MoE）。本文研究将LLM向上升级为MoE模型的可扩展性，其规律尚未充分探索。通过大量实验，我们发现描述性能随数据集规模和模型配置变化的实证缩放定律。尤其揭示，尽管扩大这些因素可提升性能，但在大计算预算下，密集模型与复用数据集之间存在新的交互项，限制了向上升级的效率。基于此，我们提供优化向上升级的建议，并确立在预算约束下，向上升级优于从零训练的具体条件。
原文摘要 · Abstract (English)
Pretraining large language models (LLMs) is resource-intensive, often requiring months of training time even with high-end GPU clusters. There are two approaches of mitigating such computational demands: reusing smaller models to train larger ones (upcycling), and training computationally efficient models like mixture-of-experts (MoE). In this paper, we study the upcycling of LLMs to MoE models, of which the scaling behavior remains underexplored. Through extensive experiments, we identify empirical scaling laws that describe how performance depends on dataset size and model configuration. Particularly, we show that, while scaling these factors improves performance, there is a novel interaction term between the dense and upcycled training dataset that limits the efficiency of upcycling at large computational budgets. Based on these findings, we provide guidance to scale upcycling, and establish conditions under which upcycling outperforms from-scratch trainings within budget constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。