用小模型初始化大模型，提速训练且不降精度。
Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization
- 用预训练小模型参数扩展生成大模型初始权重
- 大模型预训练节省超半数GPU小时（72%）
- 适合想快速部署大模型的研究者和工程师
语言模型预训练通常从随机初始化参数开始。随着模型规模扩大，其大量参数的训练变得极其耗时且昂贵。相比之下，小模型训练成本较低，但往往无法达到大模型的性能。本文探索一个新思路：能否用已训练的小模型来初始化大模型？提出HyperCloning方法，可将预训练小语言模型的参数扩展至更大模型，增加隐藏层维度，并确保大模型保留小模型的功能性。由此，大模型在训练前即具备小模型的预测能力与准确率。实验表明，使用该初始化方式可显著减少大规模语言模型预训练所需的GPU小时数（达72%），大幅降低训练成本。
原文摘要 · Abstract (English)
The pre-training phase of language models often begins with randomly initialized parameters. With the current trends in scaling models, training their large number of parameters can be extremely slow and costly. In contrast, small language models are less expensive to train, but they often cannot achieve the accuracy of large models. In this paper, we explore an intriguing idea to connect these two different regimes: Can we develop a method to initialize large language models using smaller pre-trained models? Will such initialization bring any benefits in terms of training time and final accuracy? In this paper, we introduce HyperCloning, a method that can expand the parameters of a pre-trained language model to those of a larger model with increased hidden dimensions. Our method ensures that the larger model retains the functionality of the smaller model. As a result, the larger model already inherits the predictive power and accuracy of the smaller model before the training starts. We demonstrate that training such an initialized model results in significant savings in terms of GPU hours required for pre-training large language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。