对比压缩方法在分布式训练中的抗噪能力，发现符号压缩更稳定。
Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEs
- 用随机微分方程分析两种压缩算法的噪声鲁棒性。
- 无偏量化易受重尾梯度噪声干扰，符号压缩则保持稳定。
- 提出超参数调优新规则，适用于实际分布式系统部署。
分布式学习是处理大规模模型与数据集的关键技术，但其通信开销常成为训练瓶颈。量化和稀疏化等压缩方法可缓解该问题，然而它们在语言建模中常见的大且重尾梯度噪声下的鲁棒性尚不明确。本文通过随机微分方程（SDE）分析分布式压缩SGD（DCSGD）与分布式符号SGD（DSignSGD），发现使用无偏压缩的DCSGD对噪声更为敏感，而DSignSGD即使在强噪声下仍具鲁棒性。此外，我们提出了新的超参数缩放规则，以减轻压缩导致的性能下降。这些结论在多种深度学习架构与数据集上得到验证，为分布式优化提供了实用建议。
原文摘要 · Abstract (English)
Distributed methods are essential for handling machine learning pipelines comprising large-scale models and datasets. However, their benefits often come at the cost of increased communication overhead between the central server and agents, which can become the main bottleneck, making training costly or even unfeasible in such systems. Compression methods such as quantization and sparsification can alleviate this issue. Still, their robustness to large and heavy-tailed gradient noise, a phenomenon sometimes observed in language modeling, remains poorly understood. This work addresses this gap by analyzing Distributed Compressed SGD (DCSGD) and Distributed SignSGD (DSignSGD) using stochastic differential equations (SDEs). Our results show that DCSGD with unbiased compression is more vulnerable to noise in stochastic gradients, while DSignSGD remains robust, even under large and heavy-tailed noise. Additionally, we propose new scaling rules for hyperparameter tuning to mitigate performance degradation due to compression. These findings are empirically validated across multiple deep learning architectures and datasets, providing practical recommendations for distributed optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。