arXiv:2510.21314cs.LGcs.AI2025-10被引 8

首次分析自适应优化器在浮点量化下的收敛性,解释低精度训练为何仍有效。

A Convergence Analysis of Adaptive Optimizers under Floating-point Quantization

  • 构建首个浮点量化下自适应优化器的理论框架,涵盖梯度、权重与状态量量化。
  • 证明当尾数长度随迭代次数对数增长时,收敛率接近全精度结果。
  • 揭示Adam对权重和二阶矩量化敏感,Muon则更鲁棒,适合低精度部署。

大型语言模型的快速扩展使低精度训练成为降低内存消耗、提升效率并支持更大模型与数据集的关键手段。然而,现有自适应优化器的收敛理论假设所有计算均为精确值,忽略了硬件相关的量化影响,导致低精度训练的有效性缺乏理论解释。本文首次建立自适应优化器(包括Adam和MuOn)在梯度、权重及优化器状态(如动量估计)浮点量化下的理论分析框架。在标准随机梯度假设下,我们推导了光滑非凸目标函数上的收敛速率,明确刻画了各组件量化误差对收敛的影响。结果显示,只要尾数长度随迭代次数对数级增长,两种算法的收敛速率仍可接近全精度水平。进一步分析表明,由于Adam依赖于β₂趋近1,其对权重和二阶矩量化极为敏感;而MuOn对误差控制要求更低,具有更强鲁棒性。合成与真实数据上的数值实验验证了理论结论。

原文摘要 · Abstract (English)

The rapid scaling of large language models (LLMs) has made low-precision training essential for reducing memory, improving efficiency, and enabling larger models and datasets. Existing convergence theories for adaptive optimizers, however, assume all components are exact and neglect hardware-aware quantization, leaving open the question of why low-precision training remains effective. We introduce the first theoretical framework for analyzing the convergence of adaptive optimizers, including Adam and Muon, under floating-point quantization of gradients, weights, and optimizer states (e.g., moment estimates). Within this framework, we derive convergence rates on smooth non-convex objectives under standard stochastic gradient assumptions, explicitly characterizing how quantization errors from different components affect convergence. We show that both algorithms retain rates close to their full-precision counterparts provided mantissa length scales only logarithmically with the number of iterations. Our analysis further reveals that Adam is highly sensitive to weights and second-moment quantization due to its reliance on $β_2 \to 1$, while Muon requires weaker error control and is thus potentially more robust. These results narrow the gap between empirical success and theoretical understanding of low-precision training methods. Numerical experiments on synthetic and real-world data corroborate our theory.

优化器低精度训练收敛分析浮点量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。