对比三种优化器在低预算预训练大模型中的表现
Pre-Training LLMs on a budget: A comparison of three optimizers
- 比较AdamW、Lion和Sophia三种优化器的性能
- Sophia损失最低，Lion训练最快，AdamW下游效果最好
- 适合关注模型效率与性能权衡的研究者
优化器在降低大语言模型预训练时间并提升模型性能方面起关键作用。本研究对比了三种主流优化器：标准的AdamW、通过进化搜索设计的Lion，以及二阶优化器Sophia。为提升泛化能力，采用两种基础架构，并分别使用单周期和多周期训练策略，同时保持总训练词数不变。基于最大更新参数化方法和小型代理模型，针对每种架构与优化器组合独立调参。结果显示，三种优化器表现相近，但Sophia的训练与验证损失最低，Lion在训练GPU小时上最快，而AdamW在下游任务评估中表现最优。
原文摘要 · Abstract (English)
Optimizers play a decisive role in reducing pre-training times for LLMs and achieving better-performing models. In this study, we compare three major variants: the de-facto standard AdamW, the simpler Lion, developed through an evolutionary search, and the second-order optimizer Sophia. For better generalization, we train with two different base architectures and use a single- and a multiple-epoch approach while keeping the number of tokens constant. Using the Maximal Update Parametrization and smaller proxy models, we tune relevant hyperparameters separately for each combination of base architecture and optimizer. We found that while the results from all three optimizers were in approximately the same range, Sophia exhibited the lowest training and validation loss, Lion was fastest in terms of training GPU hours but AdamW led to the best downstream evaluation results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。