用交叉熵优化提升深度网络训练的超参数调优效果
Cross-Entropy Optimization for Hyperparameter Optimization in Stochastic Gradient-based Approaches to Train Deep Neural Networks
- 基于期望最大化框架，用交叉熵优化超参数
- 可显著加速收敛并提升模型泛化性能
- 适合需要自动调参的深度学习研究者
本文提出一种基于交叉熵优化的超参数调优方法，用于基于随机梯度的深度神经网络训练。学习算法的超参数对模型收敛速度、泛化性能等有显著影响。尽管某些情况下超参数可作为学习参数处理，但在多数场景中，如Adam及其变体这类随机优化算法的超参数通常固定为常数或单调变化。本文在期望最大化（EM）框架下深入分析该方法，所提出的交叉熵超参数优化算法（CEHPO）可推广至深度学习中的其他优化问题。我们希望该方法能为机器学习及更广泛领域的优化问题提供新视角与启示。
原文摘要 · Abstract (English)
In this paper, we present a cross-entropy optimization method for hyperparameter optimization in stochastic gradient-based approaches to train deep neural networks. The value of a hyperparameter of a learning algorithm often has great impact on the performance of a model such as the convergence speed, the generalization performance metrics, etc. While in some cases the hyperparameters of a learning algorithm can be part of learning parameters, in other scenarios the hyperparameters of a stochastic optimization algorithm such as Adam [5] and its variants are either fixed as a constant or are kept changing in a monotonic way over time. We give an in-depth analysis of the presented method in the framework of expectation maximization (EM). The presented algorithm of cross-entropy optimization for hyperparameter optimization of a learning algorithm (CEHPO) can be equally applicable to other areas of optimization problems in deep learning. We hope that the presented methods can provide different perspectives and offer some insights for optimization problems in different areas of machine learning and beyond.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。