arXiv:2605.13434cs.LGcs.DC2026-05被引 2

解决异构计算下异步梯度下降的偏差问题,让快慢设备公平贡献。

Rescaled Asynchronous SGD: Optimal Distributed Optimization under Data and System Heterogeneity

  • 按计算时间调整学习率,使各设备贡献均衡。
  • 理论证明收敛到正确全局目标,时间复杂度达最优下界。
  • 无需额外缓存或同步,适合真实异构分布式训练场景。

异步随机梯度下降(ASGD)通过允许服务器在任意梯度到达时更新模型,有效利用异构计算资源。但传统ASGD对所有梯度赋予相同权重,在数据分布异构时导致偏差:计算快的设备贡献更多更新,使优化偏向局部目标的频率加权平均而非期望的全局目标。现有方法常引入收集阶段、缓冲或额外内存来修复此问题。本文证明这并非必要:仅保持标准ASGD机制,通过按各设备计算时间反比调整步长,即可使每个设备在周期内贡献相同的总学习率。在非凸设定下,于光滑性和有界异构性假设下,证明所提方法——重缩放异步SGD(Rescaled ASGD)收敛至正确全局目标的驻点。其时间复杂度在主导项上达到已知下界,而延迟与数据异构的影响仅存在于低阶项。实验验证该方法能收敛至正确目标,且性能媲美前沿基线。

原文摘要 · Abstract (English)

Asynchronous stochastic gradient descent (ASGD) is a standard way to exploit heterogeneous compute resources in distributed learning: instead of forcing fast workers to wait for slow ones, the server updates the model whenever a gradient arrives. Vanilla ASGD applies each arriving gradient with the same weight. When local data distributions are heterogeneous, this becomes problematic: faster workers contribute more updates, and we show theoretically that the method is biased toward a frequency-weighted average of the local objectives rather than the desired global objective. Existing remedies typically move away from the simple ASGD template by introducing gathering phases, buffering, or extra memory. We show that this is unnecessary. Keeping the standard ASGD mechanism, we recover the correct objective by rescaling worker-specific stepsizes in proportion to their computation times, so that each worker contributes the same aggregate learning rate over a cycle. In the non-convex setting, under smoothness and bounded heterogeneity assumptions, we prove that the resulting method, Rescaled ASGD, converges to stationary points of the correct global objective in the fixed-computation model. Its time complexity matches the known lower bound in the leading term, while the effects of staleness and data heterogeneity appear only in lower-order terms. Experiments confirm that the method converges to the correct objective and is competitive with state-of-the-art baselines.

分布式优化异步学习异构系统梯度下降

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。