解决分布式训练中计算时间不均导致的效率下降问题
MindFlayer SGD: Efficient Parallel SGD in the Presence of Heterogeneous and Random Worker Compute Times
- 设计新算法应对随机且异构的计算延迟
- 在重尾噪声环境下性能显著优于现有方法
- 适合大规模分布式学习场景使用
我们研究在多个并行工作节点环境下，最小化光滑非凸函数期望值的问题。这些工作节点可计算随机梯度，但其计算时间存在任意异构和随机性，严重制约了现有并行随机梯度下降（SGD）方法的性能。尽管部分并行SGD算法在确定性异构延迟下表现最优，但在计算时间随机的情况下效果显著下降——这正是其设计未明确考虑的情形。为此，我们提出MindFlayer SGD，一种专为处理随机且异构计算时间而设计的新式并行SGD方法。通过理论分析与实验验证，我们证明该方法在各类环境中持续优于现有基线，尤其在重尾噪声条件下表现出更强鲁棒性与可扩展性，是大规模分布式学习任务的理想选择。
原文摘要 · Abstract (English)
We investigate the problem of minimizing the expectation of smooth nonconvex functions in a distributed setting with multiple parallel workers that are able to compute stochastic gradients. A significant challenge in this context is the presence of arbitrarily heterogeneous and stochastic compute times among workers, which can severely degrade the performance of existing parallel stochastic gradient descent (SGD) methods. While some parallel SGD algorithms achieve optimal performance under deterministic but heterogeneous delays, their effectiveness diminishes when compute times are random - a scenario not explicitly addressed in their design. To bridge this gap, we introduce MindFlayer SGD, a novel parallel SGD method specifically designed to handle stochastic and heterogeneous compute times. Through theoretical analysis and empirical evaluation, we demonstrate that MindFlayer SGD consistently outperforms existing baselines, particularly in environments with heavy-tailed noise. Our results highlight its robustness and scalability, making it a compelling choice for large-scale distributed learning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。