通过显式与隐式渐进优化提升神经网络训练效果
Explicit and Implicit Graduated Optimization in Deep Neural Networks
- 显式优化用噪声平滑目标函数并逐步精炼解
- 隐式优化利用SGD动量的随机噪声实现平滑，实验验证有效
- 适用于图像分类任务，尤其在ResNet上表现优异
渐进优化是一种全局优化技术，通过添加噪声平滑多模态非凸目标函数，并逐步精炼解。本文实验评估了基于先前研究最优噪声调度的显式渐进优化算法性能，并讨论其局限性。采用传统基准函数和现代神经网络架构的实证损失函数进行测试。此外，本文扩展了基于随机梯度下降（SGD）中固有噪声可隐式平滑目标函数这一事实的隐式渐进优化算法，将其应用于带动量的SGD，分析其收敛性，并在使用ResNet架构的图像分类任务上通过实验验证了其有效性。
原文摘要 · Abstract (English)
Graduated optimization is a global optimization technique that is used to minimize a multimodal nonconvex function by smoothing the objective function with noise and gradually refining the solution. This paper experimentally evaluates the performance of the explicit graduated optimization algorithm with an optimal noise scheduling derived from a previous study and discusses its limitations. It uses traditional benchmark functions and empirical loss functions for modern neural network architectures for evaluating. In addition, this paper extends the implicit graduated optimization algorithm, which is based on the fact that stochastic noise in the optimization process of SGD implicitly smooths the objective function, to SGD with momentum, analyzes its convergence, and demonstrates its effectiveness through experiments on image classification tasks with ResNet architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。