LQ-SGD通过低秩量化压缩梯度，大幅降低通信开销且保持训练效率与模型精度。
Trustworthy Efficient Communication for Distributed Learning using LQ-SGD Algorithm
- 结合低秩近似与对数量化，压缩梯度传输数据量
- 在ResNet-50上实现97%的通信压缩率，训练收敛速度不变
- 相比传统SGD更抗梯度反演攻击，适合安全敏感的分布式系统
我们提出LQ-SGD（低秩量化随机梯度下降），一种面向分布式训练的高效通信梯度压缩算法。LQ-SGD在PowerSGD基础上引入低秩近似和对数量化技术，显著降低通信开销，同时保证训练收敛速度和模型精度。此外，基于压缩的LQ-SGD及其他方法相比传统SGD展现出更强的梯度反演抵抗能力，在保障优化效率的同时提升系统安全性，为分布式学习系统提供更鲁棒、高效的优化路径。
原文摘要 · Abstract (English)
We propose LQ-SGD (Low-Rank Quantized Stochastic Gradient Descent), an efficient communication gradient compression algorithm designed for distributed training. LQ-SGD further develops on the basis of PowerSGD by incorporating the low-rank approximation and log-quantization techniques, which drastically reduce the communication overhead, while still ensuring the convergence speed of training and model accuracy. In addition, LQ-SGD and other compression-based methods show stronger resistance to gradient inversion than traditional SGD, providing a more robust and efficient optimization path for distributed learning systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。