通过工人分歧揭示局部SGD中的陡峭方向，无需计算海森矩阵即可定位关键优化路径。
Worker Disagreement Reveals Sharp Directions in Local SGD
- 利用各节点平均梯度差的协方差，无须海森矩阵估算主方向。
- 在多层感知机、卷积网络和Transformer上，该方法捕获了主特征空间中大部分梯度分量。
- 适合关注高效优化方向识别的研究者，尤其适用于分布式训练场景。
深度神经网络训练常呈现高度各向异性的损失几何结构，少数陡峭主导的海森方向与大量平坦的主体区域共存。梯度往往过度对齐于这些主导方向，但稳定优化通常需沿更平坦的方向推进。直接基于海森矩阵的方法虽能估计主导子空间，但开销巨大。本文发现标准局部随机梯度下降（Local SGD）可通过工人之间的分歧暴露此几何结构。理论表明，工人平均梯度差的协方差受随机梯度噪声和海森曲率共同影响，导致工人在曲率敏感的陡峭方向上产生显著分歧。因此，工人平均差值构成了一种廉价的、无需海森矩阵的主导子空间估计器。在MLP、CNN和Transformer上的实验显示，由工人平均差形成的子空间可捕捉到相当比例位于主导海森特征空间内的梯度分量。
原文摘要 · Abstract (English)
Deep neural network training often exhibits highly anisotropic loss geometry, where a few sharp dominant Hessian directions coexist with a large flatter bulk. Gradients tend to align disproportionately with these dominant directions, although stable progress often requires movement through flatter bulk directions. Estimating the dominant subspace is therefore useful but costly with direct Hessian-based methods. We show that standard Local SGD exposes this geometry through worker disagreement. We theoretically show that the worker-average gap covariance is shaped by stochastic-gradient noise and Hessian curvature, causing workers to disagree along sharp, curvature-sensitive directions. Thus, worker-average gaps provide a cheap Hessian-free estimator of the dominant subspace. Experiments on MLPs, CNNs, and Transformers show that subspaces formed by worker-average gaps capture a substantial fraction of the gradient component lying in the dominant Hessian eigenspace.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。