用输出比值替代差值,实现单步完成神经网络训练
Multiplicative learning from observation-prediction ratios
- 基于观测与预测输出的比值进行参数更新
- 在图像分类任务中单次迭代即达最优权重
- 无需学习率调参,适合快速训练场景
梯度下降及其自适应变体依赖于加性参数更新,但通常需大量迭代和复杂的学习率调度来应对损失函数的尺度与曲率。本文提出期望反射(ER),一种基于观测输出与预测输出比值的乘法学习范式。该方法无需人工设计损失函数或调整学习率,同时保持内部一致性。将ER扩展至多层网络后,在图像分类任务中仅需一次迭代即可实现最优权重配置。进一步分析表明,ER可视为引入逆目标传播映射的改进梯度下降。这些结果表明,ER是传统优化方法的一种快速且可扩展的替代方案。
原文摘要 · Abstract (English)
Additive parameter updates, as used in gradient descent and its adaptive extensions, underpin most modern machine-learning optimization. Yet, such additive schemes often demand numerous iterations and intricate learning-rate schedules to cope with scale and curvature of loss functions. Here we introduce Expectation Reflection (ER), a multiplicative learning paradigm that updates parameters based on the ratio of observed to predicted outputs, rather than their differences. ER eliminates the need for ad hoc loss functions or learning-rate tuning while maintaining internal consistency. Extending ER to multilayer networks, we demonstrate its efficacy in image classification, achieving optimal weight determination in a single iteration. We further show that ER can be interpreted as a modified gradient descent incorporating an inverse target-propagation mapping. Together, these results position ER as a fast and scalable alternative to conventional optimization methods for neural-network training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。