系统对比LLM预训练中各类优化器，给出适用场景建议。
Benchmarking Optimizers for Large Language Model Pretraining
- 在标准预训练场景下统一评测多种优化方法
- 发现不同模型规模和批量下最优优化器各异
- 代码开源，支持未来方法严格基准测试
大型语言模型（LLMs）的快速发展伴随着众多新优化方法的涌现，声称可实现更快收敛或减少对特定超参数的依赖。然而，各方法采用的实验协议差异大，导致直接比较困难。本文在标准化的LLM预训练场景下，系统评估了近期优化技术，通过调整模型规模、批次大小和训练时长，对每种方法进行精细调优，为实践者提供适配不同场景的最佳优化器选择建议。对研究者而言，本工作揭示了未来优化研究的潜在方向。最后，我们公开所有代码与实验，确保结果完全可复现，旨在推动未来方法的严谨评估与发展。
原文摘要 · Abstract (English)
The recent development of Large Language Models (LLMs) has been accompanied by an effervescence of novel ideas and methods to better optimize the loss of deep learning models. Claims from those methods are myriad: from faster convergence to removing reliance on certain hyperparameters. However, the diverse experimental protocols used to validate these claims make direct comparisons between methods challenging. This study presents a comprehensive evaluation of recent optimization techniques across standardized LLM pretraining scenarios, systematically varying model size, batch size, and training duration. Through careful tuning of each method, we provide guidance to practitioners on which optimizer is best suited for each scenario. For researchers, our work highlights promising directions for future optimization research. Finally, by releasing our code and making all experiments fully reproducible, we hope our efforts can help the development and rigorous benchmarking of future methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。