通过多重积分过滤尖锐极小值，提升Adam优化器的泛化能力。
A Method for Enhancing Generalization of Adam by Multiple Integrations
- 在Adam中引入多重积分项，抑制优化过程中的尖锐极小值。
- 在多个基准测试中超越Adam及其变体，提升泛化性能与抗标签噪声能力。
- 适用于需要快速收敛且高泛化的深度学习任务，如图像分类与自然语言处理。
自适应矩估计（Adam）的泛化能力不足限制了其广泛应用。近期研究表明，损失曲面中的平坦极小值与更好的泛化性能密切相关。受积分操作对高频信号的滤波效应启发，本文提出多重积分Adam（MIAdam），在传统Adam中引入多重积分项，有效过滤优化过程中遇到的尖锐极小值，引导优化器趋向更平坦的区域，从而提升泛化能力。我们基于扩散理论框架为泛化改进提供理论解释，并分析多重积分项对收敛性的影响。实验结果表明，MIAdam不仅增强了模型泛化性和对标签噪声的鲁棒性，还保持了Adam的快速收敛特性，在多个前沿基准测试中表现优于Adam及其变体。
原文摘要 · Abstract (English)
The insufficient generalization of adaptive moment estimation (Adam) has hindered its broader application. Recent studies have shown that flat minima in loss landscapes are highly associated with improved generalization. Inspired by the filtering effect of integration operations on high-frequency signals, we propose multiple integral Adam (MIAdam), a novel optimizer that integrates a multiple integral term into Adam. This multiple integral term effectively filters out sharp minima encountered during optimization, guiding the optimizer towards flatter regions and thereby enhancing generalization capability. We provide a theoretical explanation for the improvement in generalization through the diffusion theory framework and analyze the impact of the multiple integral term on the optimizer's convergence. Experimental results demonstrate that MIAdam not only enhances generalization and robustness against label noise but also maintains the rapid convergence characteristic of Adam, outperforming Adam and its variants in state-of-the-art benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。