改进Adam算法，提升深度学习训练速度与泛化能力
Adaptive Moment Estimation Optimization Algorithm Using Projection Gradient for Deep Learning
- 引入投影梯度与幂次动量估计，增强优化适应性
- 在CIFAR-10/100上，VGG-16和ResNet-18均加速收敛并提升精度
- 理论证明适用于非凸问题，适合追求高效训练的工程师
深度神经网络训练困难。为加速训练并提升性能，我们提出PadamP这一新型优化算法。该算法基于尺度不变性下二阶矩的p次幂自适应估计，通过修改投影判别条件提升投影适应性，并集成至Adam类算法中。结合投影梯度与自适应动量估计，PadamP有效解决无约束非凸优化问题。本文分析了非凸情况下的收敛性，重点在于一阶与二阶动量估计系数的解耦。相比以往工作，证明更具普适性。在CIFAR-10和CIFAR-100数据集上使用VGG-16与ResNet-18的实验表明，PadamP在收敛速度与泛化能力上优于现有算法，是深度学习优化领域的有效补充。
原文摘要 · Abstract (English)
Training deep neural networks is challenging. To accelerate training and enhance performance, we propose PadamP, a novel optimization algorithm. PadamP is derived by applying the adaptive estimation of the p-th power of the second-order moments under scale invariance, enhancing projection adaptability by modifying the projection discrimination condition. It is integrated into Adam-type algorithms, accelerating training, boosting performance, and improving generalization in deep learning. Combining projected gradient benefits with adaptive moment estimation, PadamP tackles unconstrained non-convex problems. Convergence for the non-convex case is analyzed, focusing on the decoupling of first-order moment estimation coefficients and second-order moment estimation coefficients. Unlike prior work relying on , our proof generalizes the convergence theorem, enhancing practicality. Experiments using VGG-16 and ResNet-18 on CIFAR-10 and CIFAR-100 show PadamP's effectiveness, with notable performance on CIFAR-10/100, especially for VGG-16. The results demonstrate that PadamP outperforms existing algorithms in terms of convergence speed and generalization ability, making it a valuable addition to the field of deep learning optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。