用异步并行加速深度学习训练，提升效率且保证收敛
Distributed Deep Learning using Stochastic Gradient Staleness
- 采用数据并行与解耦反向传播结合的分布式训练
- 在CIFAR-10上实现更快训练速度，且理论证明可收敛到临界点
- 适合需要高效训练大规模模型的研究者和工程师
尽管深度神经网络在解决复杂任务上取得显著成功，其训练过程仍面临巨大挑战。主要障碍在于训练时间过长，尤其是高性能深度神经网络趋向更深（隐藏层更多）且需大量训练数据。为应对这些问题，本文提出一种融合数据并行与全解耦并行反向传播算法的分布式训练方法。通过多个计算单元并行运行，该方法在每轮迭代中处理更多训练数据，同时缓解反向传播常见的锁竞争问题。这些特性共同显著提升了训练效率。所提方法在特定条件下被严格证明可收敛至临界点。其有效性通过实证评估进一步验证：在CIFAR-10数据集上训练DNN完成分类任务。
原文摘要 · Abstract (English)
Despite the notable success of deep neural networks (DNNs) in solving complex tasks, the training process still remains considerable challenges. A primary obstacle is the substantial time required for training, particularly as high performing DNNs tend to become increasingly deep (characterized by a larger number of hidden layers) and require extensive training datasets. To address these challenges, this paper introduces a distributed training method that integrates two prominent strategies for accelerating deep learning: data parallelism and fully decoupled parallel backpropagation algorithm. By utilizing multiple computational units operating in parallel, the proposed approach enhances the amount of training data processed in each iteration while mitigating locking issues commonly associated with the backpropagation algorithm. These features collectively contribute to significant improvements in training efficiency. The proposed distributed training method is rigorously proven to converge to critical points under certain conditions. Its effectiveness is further demonstrated through empirical evaluations, wherein an DNN is trained to perform classification tasks on the CIFAR-10 dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。