根据训练状态动态切换优化器，加速收敛并提升泛化性能
AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization
- 通过六种梯度信号监控优化过程，自动在AdamW、SGD-M和Lion间切换
- CIFAR-100上仅需81轮即达78%准确率，比AdamW快26%，比SGD-M快43%
- 无需调参即可在8个基准上取得6项最佳，适合追求高效训练的实践者
单一优化器难以适应深度网络优化的不同阶段：自适应方法虽能处理早期噪声梯度，但易陷入平坦极小值；而带动量的SGD后期泛化性好，初期收敛慢。我们提出AOS-R（基于规则的自适应优化器切换），通过监测六种在线梯度空间信号——梯度噪声尺度（GNS）、Hutchinson曲率迹、损失停滞、更新稳定性比、梯度稳定性指数（GSI）和损失改善比（LIR）——随优化过程演化，在AdamW、SGD-M和Lion间动态切换。采用保持状态的动量传递与400步学习率桥接机制，确保每次切换不损失精度。在CIFAR-100/WRN-28x10上，AOS-R仅用81轮达到78%准确率，较AdamW（109轮）减少26%，较SGD-M（143轮）减少43%，较Lion（96轮）减少16%。在八个模型-数据集组合中，有6组取得最优准确率，平均提升0.4个百分点，相比AdamW收敛速度提升0.80倍，且仅需一套共享超参数配置。
原文摘要 · Abstract (English)
Single-optimizer training is a poor fit for the distinct phases of deep network optimization: adaptive methods handle noisy early gradients well but overshoot flat minima, while SGD with momentum generalizes better in the late phase but converges slowly early on. We introduce AOS-R (Adaptive Optimizer Switching, Rule-Based), a lightweight controller that monitors six online gradient-space signals -- gradient noise scale (GNS), Hutchinson curvature trace, loss stagnation, update stability ratio, gradient stability index (GSI), and loss improvement ratio (LIR) -- and switches among AdamW, SGD-M, and Lion as the optimization landscape evolves. State-preserving momentum transfer and a 400-step learning-rate bridge prevent accuracy degradation at every transition point. On CIFAR-100/WRN-28x10, AOS-R reaches 78% top-1 in 81 epochs -- 26% fewer than AdamW (109), 43% fewer than SGD-M (143), and 16% fewer than Lion (96). Across eight model-dataset benchmarks, AOS-R achieves best accuracy on 6 of 8 combinations with a mean +0.4 pp gain and 0.80x convergence speedup over AdamW under a single shared hyperparameter configuration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。