改进1比特梯度压缩，提升收敛速度与精度。
Enhancing SignSGD: Small-Batch Convergence Analysis and a Hybrid Switching Strategy
- 引入预签名噪声实现概率性幅度恢复，克服1比特损失
- 推导小批量收敛率，摆脱对大批次的依赖
- 动态切换策略融合SignSGD与SGD，适合资源受限训练
SignSGD将每个随机梯度坐标压缩为1比特，大幅节省内存与通信开销，但其1比特量化会丢失幅度信息，相比调优后的SGD存在泛化差距。本文从1比特量化与抖动视角重审SignSGD，提出三项改进：首先，在单峰对称梯度噪声下，基于信噪比加权平稳性度量，推导出无需大批次假设的小批量收敛速率；其次，在符号操作前注入衰减高斯噪声，作为经典抖动机制，概率性恢复硬阈值中丢失的幅度信息；第三，将SWATS策略适配至符号更新，采用基于投影的学习率校准，实现从SignSGD到SGD的平滑过渡。在ResNet-18上单工作器实验隔离优化器效应：预签名抖动在CIFAR-100上超越Adam，校准切换策略在CIFAR-10上达到92.18%测试准确率，优于纯SGD（91.38%）和带动量的纯SignSGD（90.82%）。
原文摘要 · Abstract (English)
SignSGD compresses each stochastic gradient coordinate to a single bit, offering substantial memory and communication savings, but its 1-bit quantization removes magnitude information and is known to leave a generalization gap relative to well-tuned SGD. We revisit SignSGD from a 1-bit quantization and dithering perspective and contribute three improvements. First, we derive a small-batch convergence rate for SignSGD under unimodal symmetric gradient noise using a signal-to-noise weighted stationarity measure, removing the large-batch assumption of prior analyses. Second, we inject annealed Gaussian noise before the sign operator, which acts as a classical dithering mechanism and probabilistically restores magnitude information lost to hard thresholding. Third, we adapt the SWATS strategy to sign-based updates with a projection-based learning-rate calibration that smoothly transitions from SignSGD to SGD. Single-worker experiments on ResNet-18 isolate optimizer effects from communication aspects: pre-sign dithering surpasses Adam on CIFAR-100, and the calibrated switch reaches 92.18% test accuracy on CIFAR-10, outperforming both pure SGD 91.38% and pure SignSGD with momentum 90.82%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。