通过有偏采样与聚合，让慢速设备协同加速深度学习训练。
Biased Local SGD for Efficient Deep Learning on Heterogeneous Systems
- 故意引入数据采样和模型聚合的偏差，协调快慢设备
- 2个CPU+8个GPU下训练ResNet20快32倍，精度相当
- 适合资源异构场景下的高效分布式训练
大多数并行神经网络训练方法假设计算资源同质。例如，同步数据并行SGD在异构负载下存在显著同步开销，常迫使实践者仅使用最快设备（如GPU）。本文研究本地SGD在异构系统上的高效训练。结果表明，有意引入数据采样和模型聚合的偏差，可有效协调较慢的CPU与较快的GPU。大量实验显示，经过精心控制的偏差能显著加速本地SGD，且在相同迭代次数下达到甚至超过同步SGD的准确率。例如，在2个CPU和8个GPU上训练ResNet20于CIFAR-10，速度比同步SGD快32倍，精度几乎相同。这些结果为灵活利用多样计算资源提供了实用洞见。
原文摘要 · Abstract (English)
Most parallel neural network training methods assume homogeneous computing resources. For example, synchronous data-parallel SGD suffers from significant synchronization overhead under heterogeneous workloads, often forcing practitioners to rely only on the fastest devices (e.g., GPUs). In this work, we study local SGD for efficient parallel training on heterogeneous systems. We show that intentionally introducing bias in data sampling and model aggregation can effectively harmonize slower CPUs with faster GPUs. Our extensive empirical results demonstrate that a carefully controlled bias significantly accelerates local SGD while achieving comparable or even higher accuracy than synchronous SGD under the same epoch budget. For instance, our method trains ResNet20 on CIFAR-10 with 2 CPUs and 8 GPUs up to 32x faster than synchronous SGD, with nearly identical accuracy. These results provide practical insights into how to flexibly utilize diverse compute resources for deep learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。