揭示大模型训练中学习率与批量大小的缩放规律，指导未来开源模型开发。
Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss
- 基于暖启动-稳定-衰减学习率策略，研究参数与模型规模、数据量的协同变化规律。
- 发现新提出的损失缩放公式能有效捕捉欠训练与过训练两种状态下的性能表现。
- 为OpenEuroLLM系列模型提供首个可复用的训练基准和超参配置方案，适合后续研究者参考。
我们研究了在英语主导语料上预训练密集型大语言模型时，学习率与批量大小的缩放行为。除了联合优化学习率与批量大小，还分析了它们随模型容量和数据规模的边际演化，并构建了能够捕捉这些关系的模型。采用暖启动-稳定-衰减的学习率调度策略，进一步在广泛的超参数设置、模型规模和数据预算下评估学习率退火带来的收益，并检验最优学习率与批量大小在稳定阶段与衰减阶段之间的可迁移性。最后，我们刻画了损失对模型容量和数据集规模的依赖关系，评估了近期提出的显式建模二者交互作用的缩放形式。实验表明，这些方法在覆盖欠训练与过训练区域时尤为有效。本研究为未来OpenEuroLLM模型的开发建立了首个基准与缩放流程。研究中使用的全部预训练运行数据已开源。
原文摘要 · Abstract (English)
We study the scaling behavior of learning rate and batch size in pretraining dense large language models on English-prevalent corpora. Beyond scaling jointly optimal learning rates and batch sizes, we investigate their marginal evolution with model capacity and data scale and develop a model that captures these relationships. As we employ a Warmup-Stable-Decay learning rate schedule, we further investigate the gains from learning rate annealing over a broad range of hyperparameters settings, models and data budgets, and whether the optimal learning rate and batch size transfer between the stable and decay phases. Finally, we characterize the dependence of loss on model capacity and dataset size, evaluating recently proposed scaling forms that explicitly model their interaction. We find these approaches particularly effective at capturing both undertraining and overtraining regimes across our experiments. This study establishes a first baseline and scaling procedure for the development of future OpenEuroLLM models. We open-source the complete collection of pretraining runs used in this study.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。