通过渐进式训练构建模型家族，节省25%计算成本。
Efficient Construction of Model Family through Progressive Training Using Model Expansion
- 从小模型逐步扩展到大模型，避免重复训练。
- 1B到8B参数族总计算量降低约25%，性能相当。
- 调整学习率使模型行为更一致，适合资源受限场景。
随着大语言模型（LLMs）广泛应用，提供不同参数规模的模型家族已成为标准做法，以适应多样化的计算需求。传统方法中，家族内每个模型独立训练，导致计算成本随模型数量线性增长。本文提出一种基于渐进式训练的高效模型家族构建方法：将小模型逐步扩展至更大规模，形成完整模型家族。在涵盖1B至8B参数的模型家族上进行大量实验表明，该方法可将总计算成本降低约25%，同时保持与独立训练模型相当的性能。此外，通过根据模型规模动态调整最大学习率，该方法在多个指标上优于独立训练。除效率提升外，该方法还增强了不同尺寸模型间的行为一致性。
原文摘要 · Abstract (English)
As Large Language Models (LLMs) gain widespread practical application, offering model families with varying parameter sizes has become standard practice to accommodate diverse computational requirements. Traditionally, each model in the family is trained independently, incurring computational costs that scale additively with the number of models. In this work, we propose an efficient method for constructing model families via progressive training, where smaller models are incrementally expanded to larger sizes to create a complete model family. Through extensive experiments on a model family ranging from 1B to 8B parameters, we show that our approach reduces total computational cost by approximately 25% while maintaining comparable performance to independently trained models. Moreover, by strategically adjusting the maximum learning rate based on model size, our method outperforms the independent training across various metrics. Beyond these improvements, our approach also fosters greater consistency in behavior across model sizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。