Lion优化器通信开销大，Lion Cub通过量化与选择性动量同步实现5倍加速。
Lion Cub: Minimizing Communication Overhead in Distributed Lion
- 针对Lion优化器设计专用量化方法，利用符号更新特性降低通信量。
- 结合选择性动量同步，实现在保持收敛性前提下通信成本显著下降。
- 适合大规模分布式训练场景，尤其适用于慢速以太网环境的高效训练。
分布式深度学习中的通信开销是主要挑战，尤其在较慢的以太网互连环境下。尽管梯度压缩技术已用于SGD和Adam，但Lion优化器因其更新向量为符号操作输出，天然适合量化。然而，单纯压缩更新并采用多数投票等方法无法带来端到端加速，原因在于通信算法效率低及收敛性下降。本文分析了三个关键因素：优化通信方法、探索有效的量化策略、评估动量同步的必要性。研究发现，针对Lion定制的量化方法与选择性动量同步可显著降低通信开销并维持收敛性。由此提出的Lion Cub方案，在端到端训练中相比原始Lion实现最高5倍加速，凸显了Lion作为高效通信分布式训练方案的潜力。
原文摘要 · Abstract (English)
Communication overhead is a key challenge in distributed deep learning, especially on slower Ethernet interconnects, and given current hardware trends, communication is likely to become a major bottleneck. While gradient compression techniques have been explored for SGD and Adam, the Lion optimizer has the distinct advantage that its update vectors are the output of a sign operation, enabling straightforward quantization. However, simply compressing updates for communication and using techniques like majority voting fails to lead to end-to-end speedups due to inefficient communication algorithms and reduced convergence. We analyze three factors critical to distributed learning with Lion: optimizing communication methods, identifying effective quantization methods, and assessing the necessity of momentum synchronization. Our findings show that quantization techniques adapted to Lion and selective momentum synchronization can significantly reduce communication costs while maintaining convergence. We combine these into Lion Cub, which enables up to 5x speedups in end-to-end training compared to Lion. This highlights Lion's potential as a communication-efficient solution for distributed training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。