针对大模型训练中的异常梯度噪声，提出异步优化新方法。
Asynchronous Heavy-Tailed Optimization
- 基于延迟感知的学习率调度与补偿机制改进异步算法
- 理论证明收敛速度达同步算法水平，且更耐延迟
- 在图像与语言任务中更稳定、效率更高
Transformer 模型中的重尾随机梯度噪声会破坏优化过程。现有研究多集中于集中式或同步分布式场景下的处理方法，而异步优化中此类噪声的影响尚未充分探索。本文研究两种处理延迟节点的异步通信方案，提出基于延迟感知学习率调度和延迟补偿的算法改进。理论上，该方法在重尾噪声下仍能实现与同步算法相当的收敛速率，并比现有异步方法更具延迟容忍性。实验表明，该方法在图像与语言任务中均优于先前的同步与异步方法，在准确率/运行时间权衡上表现更优，且对超参数更鲁棒。
原文摘要 · Abstract (English)
Heavy-tailed stochastic gradient noise, commonly observed in transformer models, can destabilize the optimization process. Recent works mainly focus on developing and understanding approaches to address heavy-tailed noise in the centralized or distributed, synchronous setting, leaving the interactions between such noise and asynchronous optimization underexplored. In this work, we investigate two communication schemes that handle stragglers with asynchronous updates in the presence of heavy-tailed gradient noise. We propose and theoretically analyze algorithmic modifications based on delay-aware learning rate scheduling and delay compensation to enhance the performance of asynchronous algorithms. Our convergence guarantees under heavy-tailed noise match the rate of the synchronous counterparts and improve delay tolerance compared with existing asynchronous approaches. Empirically, our approaches outperform prior synchronous and asynchronous methods in terms of accuracy/runtime trade-offs and are more robust to hyperparameters in both image and language tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。