提出新噪声度量CWGD,让学习率自适应曲率,优化更稳定。
Curvature-Weighted Gradient Diversity: A Noise Measure for Geometry-Adaptive SGD Schedules

- 用海森逆平方根加权梯度多样性,捕捉方向性噪声影响。
- 在二次凸目标上,误差下限降低约50%,最终误差低20%。
- 适合追求高精度优化的科研人员,尤其对非凸问题有启发。
标准小批量随机梯度下降(SGD)用单一方差项建模梯度噪声,忽略了高曲率方向因学习率受限而噪声影响较小的事实。本文提出曲率加权梯度多样性(CWGD),通过海森矩阵的逆平方根对每样本梯度多样性加权,提供更精确的有效优化噪声代理。对于具有对角海森矩阵和各向同性噪声的强凸二次目标,我们证明基于CWGD调节的余弦学习率调度可使渐近优化误差下限降低至标准余弦退火的一半。我们实现了基于Hutchinson估计器的CWGD-Cosine方法,在二次目标下计算准确。在不同条件数、批量大小和噪声结构下,其最终优化误差始终比标准余弦退火低约20%,且开销极小。我们还纠正了一个退化曲率估计器,分析了估计器鲁棒性,并明确讨论了该方法局限性,包括非凸优化中海森矩阵过时问题。这些结果确立了CWGD作为优化噪声的原理性几何感知度量,并为推广至更一般学习问题提供了基础。
原文摘要 · Abstract (English)
The standard convergence analysis of mini-batch stochastic gradient descent (SGD) models gradient noise using a single variance term that treats all parameter directions equally, ignoring the fact that noise in high-curvature directions has less impact because learning rates are already constrained there. We introduce Curvature-Weighted Gradient Diversity (CWGD), a geometry-aware measure that weights per-sample gradient diversity by the inverse square root of the Hessian, providing a tighter proxy for the effective optimization noise. For strongly convex quadratic objectives with diagonal Hessians and isotropic noise, we prove that a CWGD-modulated cosine learning-rate schedule can reduce the asymptotic optimization error floor by up to a factor of two compared with standard cosine annealing. We implement this idea as CWGD-Cosine using a Hutchinson-based diagonal Hessian estimator that is exact for quadratic objectives. Across a range of condition numbers, batch sizes, and noise structures, CWGD-Cosine consistently achieves approximately 20% lower final optimization error than standard cosine annealing while incurring negligible overhead in the quadratic setting. We also identify and correct a degenerate curvature estimator, analyze the robustness of the proposed estimator, and explicitly discuss the limitations of the method, including Hessian staleness in non-convex optimization. These results establish CWGD as a principled geometry-aware measure of optimization noise and motivate future extensions to more general learning problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。