新优化器ArcGD通过几何相位控制步长,提升训练稳定性和精度。
Arc Gradient Descent: A Geometrically Motivated Gradient Descent-based Optimiser with Phase-Aware, User-Controlled Step Dynamics (proof-of-concept)
- 基于几何相位设计步长动态,自动调节优化方向与幅度。
- 在50,000维罗森伯克函数上优于Adam,CIFAR-10上达50.7%准确率。
- 适合需要稳定收敛和抗过拟合的深度学习任务,尤其适配复杂架构。
本文提出并评估了ArcGD优化器。首先在非凸基准函数上测试,使用随机变体的罗森伯克函数(维度2D至50,000D),对比Adam优化器,在采用有效学习率的设定下,ArcGD始终表现更优;在默认学习率下虽较慢,但多数情况达到更好最终解。其次在真实数据集上评估,对比Adam、AdamW、Lion、SGD,在8种不同结构的MLP模型上(1至5层隐含层)进行测试。在20,000次迭代时,ArcGD平均测试准确率达50.7%,高于AdamW(46.6%)、Adam(46.8%)、SGD(49.6%)、Lion(43.4%),在6个模型上胜出或持平。尽管Adam和AdamW初期收敛快,但长期训练后性能下降,而ArcGD持续提升,展现良好泛化能力与抗过拟合特性,无需早停调参。几何压力测试与主流深度学习基准均表现优异,表明其广泛适用性。此外,弧度限制变体与动量增强版可还原符号动量更新,揭示了ArcGD相位结构与Lion优化器核心机制的明确联系。
原文摘要 · Abstract (English)
The paper presents the formulation, implementation, and evaluation of the ArcGD optimiser. The evaluation is conducted initially on a non-convex benchmark function and subsequently on a real-world ML dataset. The initial comparative study using the Adam optimiser is conducted on a stochastic variant of the highly non-convex and notoriously challenging Rosenbrock function, renowned for its narrow, curved valley, across dimensions ranging from 2D to 1000D and an extreme case of 50,000D. Two configurations were evaluated to eliminate learning-rate bias: (i) both using ArcGD's effective learning rate and (ii) both using Adam's default learning rate. ArcGD consistently outperformed Adam under the first setting and, although slower under the second, achieved superior final solutions in most cases. In the second evaluation, ArcGD is evaluated against state-of-the-art optimizers (Adam, AdamW, Lion, SGD) on the CIFAR-10 image classification dataset across 8 diverse MLP architectures ranging from 1 to 5 hidden layers. ArcGD achieved the highest average test accuracy (50.7%) at 20,000 iterations, outperforming AdamW (46.6%), Adam (46.8%), SGD (49.6%), and Lion (43.4%), winning or tying on 6 of 8 architectures. Notably, while Adam and AdamW showed strong early convergence at 5,000 iterations, but regressed with extended training, whereas ArcGD continued improving, demonstrating generalization and resistance to overfitting without requiring early stopping tuning. Strong performance on geometric stress tests and standard deep-learning benchmarks indicates broad applicability, highlighting the need for further exploration. Moreover, it is also shown that both a limiting variant of ArcGD and a momentum augmented ArcGD, recover sign-based momentum updates, revealing a clear conceptual link between ArcGD's phase structure and the core mechanism of the Lion Optimiser.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。