研究不同优化器对大模型训练的影响，提出更稳定的缩放定律。
Towards Robust Scaling Laws for Optimizers
- 用统一框架分析多种优化器的缩放行为，避免参数高度相关。
- 提出共享指数、仅调整缩放因子的新定律，可直接比较优化器性能。
- 理论证明经典缩放规律源于损失分解，适用于凸二次任务。
大型语言模型预训练质量受计算预算和优化算法选择的影响。虽然经验缩放定律广泛用于预测模型规模与训练数据增长时的损失表现，但几乎所有现有研究都固定使用单一优化器（通常为AdamW）。与此同时，新一代优化器（如Muon、Shampoo、SOAP）承诺更快更稳定的收敛，但其与模型和数据缩放的关系尚未被充分理解。本文研究了不同优化器下的缩放规律。实证结果表明：1）针对每个优化器分别拟合Chinchilla风格缩放定律会导致参数高度共线且病态；2）我们提出一种更稳健的缩放定律，共享幂律指数并引入优化器特异的缩放因子，实现优化器间的直接比较；3）针对凸二次目标的代理任务，我们进行理论分析，证明Chinchilla式缩放定律自然源于损失分解为不可约误差、近似误差和优化误差三部分。
原文摘要 · Abstract (English)
The quality of Large Language Model (LLM) pretraining depends on multiple factors, including the compute budget and the choice of optimization algorithm. Empirical scaling laws are widely used to predict loss as model size and training data grow, however, almost all existing studies fix the optimizer (typically AdamW). At the same time, a new generation of optimizers (e.g., Muon, Shampoo, SOAP) promises faster and more stable convergence, but their relationship with model and data scaling is not yet well understood. In this work, we study scaling laws across different optimizers. Empirically, we show that 1) separate Chinchilla-style scaling laws for each optimizer are ill-conditioned and have highly correlated parameters. Instead, 2) we propose a more robust law with shared power-law exponents and optimizer-specific rescaling factors, which enable direct comparison between optimizers. Finally, 3) we provide a theoretical analysis of gradient-based methods for the proxy task of a convex quadratic objective, demonstrating that Chinchilla-style scaling laws emerge naturally as a result of loss decomposition into irreducible, approximation, and optimization errors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。