用双贝叶斯机制自动推导最优学习率，提升模型训练效果
Training Neural Networks with Optimal Double-Bayesian Learning
- 构建双贝叶斯框架，通过对抗性概率推演确定学习率
- 在分类、分割、检测任务中验证理论学习率的有效性
- 适合关注优化策略与模型泛化能力的研究者
梯度下降的反向传播是机器学习中神经网络常用优化方法。然而，寻找能引导训练的最优超参数仍具挑战性。尽管选择合适参数对避免过拟合和获得无偏结果至关重要，但这一选择仍主要依赖经验与试错。本文提出一种针对随机梯度下降关键参数——学习率的新概率框架。该框架将经典贝叶斯统计拓展为包含两个对立贝叶斯过程的双贝叶斯决策机制，可从中推导出理论最优学习率，并用于梯度下降。在多种分类、分割与检测任务上的实验验证了该理论学习率的实用性。论文还探讨了所提双贝叶斯框架对网络训练及模型性能的影响。
原文摘要 · Abstract (English)
Backpropagation with gradient descent is a common optimization strategy employed by most neural network architectures in machine learning. However, finding optimal hyperparameters to guide training has proven challenging. While it is widely acknowledged that selecting appropriate parameters is crucial for avoiding overfitting and achieving unbiased outcomes, this choice remains largely based on empirical experiments and experience. This paper presents a new probabilistic framework for the learning rate, a key parameter in stochastic gradient descent. The framework develops classic Bayesian statistics into a double-Bayesian decision mechanism involving two antagonistic Bayesian processes. A theoretically optimal learning rate can be derived from these two processes and used for stochastic gradient descent. Experiments across various classification, segmentation, and detection tasks corroborate the practical significance of the theoretically derived learning rate. The paper also discusses the ramifications of the proposed double-Bayesian framework for network training and model performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。