越早训练的模型，后续继续预训练收益越低。
Reusing Overtrained Language Models Saturates Scaling
- 用已有模型继续预训练，效果随初始训练量增加而递减。
- 每多训练10倍初始参数，后续收益下降约10%。
- 适合想节省成本但不盲目追大模型的研究者。
将已预训练的基础模型用于进一步预训练（如持续预训练或模型扩展），可显著降低从头训练语言模型的成本。然而，当基础模型已过度训练时，其有效性尚不明确。本文通过实证研究发现，模型复用的缩放效率呈可预测的衰减：随着基础模型预训练数据量增加，第二阶段训练的缩放指数以对数方式下降。第一阶段与第二阶段训练数据量的联合影响可通过一个简单缩放律精确建模。该饱和现象揭示了多阶段预训练中的根本权衡——基础模型预训练越充分，后续预训练带来的增益越小。研究结果为高效语言模型训练提供了实践指导，并对过度训练模型的复用提出了重要警示。
原文摘要 · Abstract (English)
Reusing pretrained base models for further pretraining, such as continual pretraining or model growth, is promising at reducing the cost of training language models from scratch. However, the effectiveness remains unclear, especially when applied to overtrained base models. In this work, we empirically study the scaling properties of model reuse and find that the scaling efficiency diminishes in a predictable manner: The scaling exponent with respect to second-stage training tokens decreases logarithmically with the number of tokens used to pretrain the base model. The joint dependence on first- and second-stage tokens is accurately modeled by a simple scaling law. Such saturation effect reveals a fundamental trade-off in multi-stage pretraining strategies: the more extensively a base model is pretrained, the less benefit additional pretraining provides. Our findings provide practical insights for efficient language model training and raise important considerations for the reuse of overtrained models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。