通过插值与噪声正则化提升SGD收敛速度与泛化能力
Randomness and Interpolation Improve Gradient Descent
- 用二阶牛顿插值加速梯度下降
- 在CIFAR数据集上提升模型收敛速度与泛化性能
- 适合追求高效训练与抗过拟合的深度学习研究者
基于随机梯度下降（SGD），本文提出两种优化器：插值加速梯度下降（IAGD）和噪声正则化随机梯度下降（NRSGD）。IAGD利用梯度在迭代间的相关性，采用二阶牛顿插值方法加速训练收敛。为防止过拟合，NRSGD引入可控噪声对梯度进行正则化。在CIFAR-10和CIFAR-100数据集上，对多种卷积神经网络（CNN）进行了对比实验，评估IAGD与NRSGD相较于Keras中经典优化器的表现。结果表明，这两种改进方法在提升收敛效率与泛化能力方面具有显著潜力。
原文摘要 · Abstract (English)
Based on Stochastic Gradient Descent (SGD), the paper introduces two optimizers, named Interpolational Accelerating Gradient Descent (IAGD) as well as Noise-Regularized Stochastic Gradient Descent (NRSGD). IAGD leverages second-order Newton Interpolation to expedite the convergence process during training, assuming relevancy in gradients between iterations. To avoid over-fitting, NRSGD incorporates a noise regularization technique that introduces controlled noise to the gradients during the optimization process. Comparative experiments of this research are conducted on the CIFAR-10, and CIFAR-100 datasets, benchmarking different CNNs(Convolutional Neural Networks) with IAGD and NRSGD against classical optimizers in Keras Package. Results demonstrate the potential of those two viable improvement methods in SGD, implicating the effectiveness of the advancements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。