提出无需学习率的优化器,让参数更新始终沿目标函数曲面走。
Geodesic Gradient Descent: A Generic and Learning-rate-free Optimizer on Objective Function-induced Manifolds
- 用n维球面近似目标函数曲面局部,通过测地线保持更新轨迹在曲面上。
- 最大步长为球面弧长的1/4,自动消除学习率需求,实验显示性能显著提升。
- 适合追求高精度优化、对学习率敏感的深度学习任务,尤其适用于复杂曲面场景。
欧几里得梯度下降难以捕捉目标函数诱导超曲面的几何结构,易使更新轨迹偏离曲面;黎曼梯度下降虽可解决此问题,但难以用单一经典流形表示复杂超曲面。本文提出测地线梯度下降(GGD),一种通用且无需学习率的黎曼梯度下降算法。每轮迭代中,GGD使用n维球面近似目标函数诱导超曲面的局部邻域,将欧氏梯度导出的切向量投影至球面形成测地线,确保更新轨迹始终位于曲面上。参数更新基于测地线终点完成。GGD中梯度最大步长等于n维球面弧长的四分之一,从而无需设置学习率。实验结果表明,在Burgers数据集上,全连接网络的测试均方误差降低35.79%至48.76%;在MNIST数据集上,卷积神经网络的交叉熵损失降低3.14%至11.59%,均优于经典Adam算法。
原文摘要 · Abstract (English)
Euclidean gradient descent algorithms barely capture the geometry of objective function-induced hypersurfaces and risk driving update trajectories off the hypersurfaces. Riemannian gradient descent algorithms address these issues but fail to represent complex hypersurfaces via a single classic manifold. We propose geodesic gradient descent (GGD), a generic and learning-rate-free Riemannian gradient descent algorithm. At each iteration, GGD uses an n-dimensional sphere to approximate a local neighborhood on the objective function-induced hypersurface, adapting to arbitrarily complex geometries. A tangent vector derived from the Euclidean gradient is projected onto the sphere to form a geodesic, ensuring the update trajectory stays on the hypersurface. Parameter updates are performed using the endpoint of the geodesic. The maximum step size of the gradient in GGD is equal to a quarter of the arc length on the n-dimensional sphere, thus eliminating the need for a learning rate. Experimental results show that compared with the classic Adam algorithm, GGD achieves test MSE reductions ranging from 35.79% to 48.76% for fully connected networks on the Burgers' dataset, and cross-entropy loss reductions ranging from 3.14% to 11.59% for convolutional neural networks on the MNIST dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。