提出新优化器GONO，用梯度方向一致性提升训练效率
Directional Consistency as a Complementary Optimization Signal: The GONO Framework
- 根据梯度方向一致性动态调整动量，增强收敛稳定性
- 在多个数据集上性能持平AdamW，且能精准检测训练震荡
- 适合需要稳定优化的深度学习场景，尤其对收敛判断敏感任务
我们揭示并形式化了深度学习优化中一个被忽视的现象：梯度方向一致性与损失下降可解耦。优化器可能表现出接近完美的方向一致性（cc_t → 1，通过连续梯度余弦相似度衡量），但损失仍高或下降缓慢。这表明现有优化器如Adam、SGD和RMSprop缺乏显式机制利用梯度方向的时间一致性，仅依赖幅值信号，难以区分平台期、鞍点与真正收敛。为此，我们提出GONO（Gradient-Oriented Norm-Adaptive Optimizer），根据cc_t动态调整Adam的动量系数beta_1：在方向一致时增强动量，在振荡时抑制动量。我们证明GONO保持Adam的O(1/sqrt(T))收敛速率，且当信号无信息时退化为Adam。实验显示，cc_t在振荡检测中达到F1=1.00（对比梯度幅值为0.45）；GONO在MNIST（98.15%）、CIFAR-10（43.14%）和ResNet-18（75.44%）上与AdamW相当，确立方向一致性为理论可靠、实践可行的优化信号。
原文摘要 · Abstract (English)
We identify and formalize an underexplored phenomenon in deep learning optimization: directional alignment and loss convergence can be decoupled. An optimizer can exhibit near-perfect directional consistency (cc_t -> 1, measured via consecutive gradient cosine similarity) while the loss remains high or decreases slowly. This observation reveals that existing optimizers such as Adam, SGD, and RMSprop lack explicit mechanisms to exploit temporal consistency in gradient directions, relying instead on magnitude-based signals that fail to distinguish plateaus, saddle points, and genuine convergence. Motivated by this, we introduce GONO (Gradient-Oriented Norm-Adaptive Optimizer), which adapts Adam's momentum coefficient beta_1 based on cc_t: amplifying momentum under directional consistency and suppressing it during oscillation. We prove GONO matches Adam's O(1/sqrt(T)) convergence rate and reduces exactly to Adam when the signal is uninformative. Empirically, cc_t achieves oscillation detection with F1=1.00 (vs. 0.45 for gradient norm), and GONO remains competitive with AdamW on MNIST (98.15%), CIFAR-10 (43.14%), and ResNet-18 (75.44%), establishing directional alignment as a theoretically grounded, practically actionable optimization signal. Code: https://github.com/victordaniel/gono-optimizer
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。