通过过滤噪声方向提升SGD信号质量，显著改善梯度压缩效果。
DOME: Improving Signal-to-Noise in Stochastic Gradient Descent via Sharp-Direction Subspace Filtering
- 用梯度协方差识别噪声子空间，无需计算黑塞矩阵。
- 移除该子空间后优化性能几乎不变，但梯度信噪比提升30%以上。
- 适合对梯度精度敏感的应用，如模型压缩与分布式训练。
深度神经网络的随机梯度在优化过程中呈现强相关性，且常对齐于黑塞矩阵中少数异常特征值对应的特征向量。已有研究显示，将梯度投影出该异常子空间对优化影响甚微，尽管其捕获了大量梯度方差。由于黑塞矩阵计算不可行，本文提出基于梯度协方差的一阶原则性表征，实现在线高效估计该冗余子空间。实验表明，移除该子空间对优化影响极小，却显著提升梯度信噪比，尤其适用于对梯度信噪比敏感的任务，如梯度压缩。
原文摘要 · Abstract (English)
Stochastic gradients for deep neural networks exhibit strong correlations along the optimization trajectory, and are often aligned with a small set of Hessian eigenvectors associated with outlier eigenvalues. Recent work shows that projecting gradients away from this Hessian outlier subspace has little impact on optimization, despite capturing a large fraction of gradient variability. Since computing the Hessian is intractable in practice, we introduce a principled first-order characterization of the nuisance subspace based on the covariance of stochastic gradients, and propose an efficient method to estimate it online. We show that removing this subspace also has little impact on optimization, and yields practical benefits for applications sensitive to gradient signal-to-noise ratio such as gradient compression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。