新优化器融合多种技巧，训练更快且泛化更好。
Frankenstein Optimizer: Harnessing the Potential by Revisiting Optimization Tricks
- 动态调整一阶二阶动量系数，保持稳定学习过程。
- 在图像、语言等多领域实验中收敛更快、泛化更强。
- 通过可视化分析揭示自适应算法的内在机制。
基于梯度的优化推动了现代深度神经网络在众多应用中的卓越表现。自适应算法因快速收敛而加速训练，但难以稳定找到‘平坦极小值’，导致泛化性能不如随机梯度下降（SGD）。本文通过重新审视各类自适应算法的机制，提出名为Frankenstein的新优化器，融合其优势。该优化器根据当前状态动态调整一阶和二阶动量系数，直接维持一致的学习动态，并迅速响应突变梯度。在计算机视觉、自然语言处理、少样本学习及科学模拟等多个研究领域进行的大量实验表明，Frankenstein在收敛速度和泛化性能上均优于现有自适应算法与SGD。此外，本研究通过中心核对齐分析与损失曲面可视化，深化了对自适应算法本质的理解。代码已开源：https://github.com/acctouhou/Frankenstein_optimizer。
原文摘要 · Abstract (English)
Gradient-based optimization drives the unprecedented performance of modern deep neural network models across diverse applications. Adaptive algorithms have accelerated neural network training due to their rapid convergence rates; however, they struggle to find ``flat minima" reliably, resulting in suboptimal generalization compared to stochastic gradient descent (SGD). By revisiting various adaptive algorithms' mechanisms, we propose the Frankenstein optimizer, which combines their advantages. The proposed Frankenstein dynamically adjusts first- and second-momentum coefficients according to the optimizer's current state to directly maintain consistent learning dynamics and immediately reflect sudden gradient changes. Extensive experiments across several research domains such as computer vision, natural language processing, few-shot learning, and scientific simulations show that Frankenstein surpasses existing adaptive algorithms and SGD empirically regarding convergence speed and generalization performance. Furthermore, this research deepens our understanding of adaptive algorithms through centered kernel alignment analysis and loss landscape visualization during the learning process. Code is available at https://github.com/acctouhou/Frankenstein_optimizer
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。