提出自适应梯度聚合方法，提升大规模分布式训练效率。
Adaptive Consensus Gradients Aggregation for Scaled Distributed Training
- 基于子空间优化设计加权梯度聚合策略
- 在MLPerf多个任务上优于传统平均法，收敛更快
- 适合大规模分布式训练场景，通信与计算开销极低
分布式机器学习已成为在海量数据上训练大型模型的关键范式。本文研究了在通信受限的同步并行计算环境下深度学习的随机优化问题。尽管平均分布梯度是最常用的梯度估计方法，但其是否最优仍存疑问。我们从子空间优化视角分析梯度聚合过程，将聚合问题建模为面向目标的子空间优化问题，推导出由子空间系数引导的高效梯度加权方案，并引入子空间动量以加速收敛，同时保持聚合的统计无偏性。该方法在多个MLPerf任务上表现优于广泛使用的梯度平均法，且在通信和计算复杂度上均保持极低开销。
原文摘要 · Abstract (English)
Distributed machine learning has recently become a critical paradigm for training large models on vast datasets. We examine the stochastic optimization problem for deep learning within synchronous parallel computing environments under communication constraints. While averaging distributed gradients is the most widely used method for gradient estimation, whether this is the optimal strategy remains an open question. In this work, we analyze the distributed gradient aggregation process through the lens of subspace optimization. By formulating the aggregation problem as an objective-aware subspace optimization problem, we derive an efficient weighting scheme for gradients, guided by subspace coefficients. We further introduce subspace momentum to accelerate convergence while maintaining statistical unbiasedness in the aggregation. Our method demonstrates improved performance over the ubiquitous gradient averaging on multiple MLPerf tasks while remaining extremely efficient in both communicational and computational complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。