用物理启发方法训练神经网络,效果优于传统优化法。
Simmering: Sufficient is better than optimal for training neural networks
- 以物理系统为灵感,不追求最优解,只求足够好
- 在分类与回归任务中有效纠正Adam的过拟合问题
- 适合追求稳定性能而非极致精度的研究者
众多神经网络训练方法依赖优化,却需人为调整以保证有效性,暗示基于优化的训练可能存在问题。过拟合现象尤为凸显其缺陷,单纯优化易导致虚假结果。近年来,神经网络在建模物理过程方面取得成功,促使研究转向将神经网络本身视为物理系统。本文提出一种名为simmering的物理启发训练方法,不追求参数最优,而是生成‘足够好’的权重与偏置,反而在多个任务中超越主流优化方法。实验表明,simmering能修正Adam导致的过拟合,并从初始阶段避免过拟合。结果挑战了以优化为核心的训练范式,结合信息几何论证,指出存在一类不以优化为基础的充分性训练算法。
原文摘要 · Abstract (English)
The broad range of neural network training techniques that invoke optimization but rely on ad hoc modification for validity suggests that optimization-based training is misguided. Shortcomings of optimization-based training are brought to particularly strong relief by the problem of overfitting, where naive optimization produces spurious outcomes. The broad success of neural networks for modelling physical processes has prompted advances that are based on inverting the direction of investigation and treating neural networks as if they were physical systems in their own right. These successes raise the question of whether broader, physical perspectives could motivate the construction of improved training algorithms. Here, we introduce simmering, a physics-based method that trains neural networks to generate weights and biases that are merely ``good enough'', but which, paradoxically, outperforms leading optimization-based approaches. Using classification and regression examples we show that simmering corrects neural networks that are overfit by Adam, and show that simmering avoids overfitting if deployed from the outset. Our results question optimization as a paradigm for neural network training, and leverage information-geometric arguments to point to the existence of classes of sufficient training algorithms that do not take optimization as their starting point.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。