提出新方法让异步训练更高效，避免复杂样本被忽略
Bringing Order to Asynchronous SGD: Towards Optimality under Data-Dependent Delays with Momentum
- 用动量机制保留延迟梯度信息，不丢弃或削弱
- 首次实现数据相关延迟下的最优收敛速度
- 适合需要高效分布式训练的科研与工程人员
异步随机梯度下降（Asynchronous SGD）可实现可扩展的分布式训练，但面临梯度过时问题。现有缓解策略如自适应学习率和过时感知过滤通常会衰减或丢弃延迟梯度，引入系统性偏差：简单或处理快的样本更新被过度代表，而复杂样本的梯度则被延迟或抑制。相比之下，以往基于Lipschitz假设的数据依赖延迟方法仅获得次优收敛率，且未涵盖光滑凸情形。本文提出一种基于动量的异步框架，旨在保留延迟梯度的信息同时减轻过时影响。我们首次在光滑凸与非凸设置下，为数据依赖延迟建立了最优收敛速率，提供了标准假设下异步优化的新结果。此外，我们推导出鲁棒的学习率调度策略，简化了实际中的超参数调优。
原文摘要 · Abstract (English)
Asynchronous stochastic gradient descent (SGD) enables scalable distributed training but suffers from gradient staleness. Existing mitigation strategies, such as delay-adaptive learning rates and staleness-aware filtering, typically attenuate or discard delayed gradients, introducing systematic bias: updates from simpler or faster-to-process samples are overrepresented, while gradients from more complex samples are delayed or suppressed. In contrast, prior approaches to data-dependent delays rely on a Lipschitz assumption that yields suboptimal rates or leave the smooth, convex case unaddressed. We propose a momentum-based asynchronous framework designed to preserve information from delayed gradients while mitigating the effects of staleness. We establish the first optimal convergence rates for data-dependent delays in both convex and non-convex smooth setups, providing a new result for asynchronous optimization under standard assumptions. Additionally, we derive robust learning-rate schedules that simplify hyperparameter tuning in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。