根据训练波动性自适应调整学习率，提升模型泛化能力
An Adaptive Volatility-based Learning Rate Scheduler
- 基于波动率比值动态调节学习率，探索与稳定兼顾
- 在CIFAR-100上使ResNet-18/34准确率分别提升1.4%/1.3%
- 找到更平坦的损失极小值，适合追求泛化的深度学习研究
有效的学习率调度对训练深度神经网络至关重要。然而，现有的预定义和自适应调度器仍可能导致次优泛化。本文提出VolSched，一种受随机过程（如几何布朗运动）中波动性启发的新型自适应学习率调度器，通过计算长期与短期精度波动率的比率，动态调整学习率：在陷入平台期时提高学习率以加速逃离，在趋于稳定时降低学习率以增强收敛。我们在标准数据增强流程下，将VolSched与强基线对比，在CIFAR-100数据集上验证其性能。结果表明，搭配ResNet-18和ResNet-34时，分别实现1.4和1.3个百分点的顶部1准确率提升。损失曲线分析显示，VolSched延长了探索阶段。对海森矩阵的定量分析进一步表明，该方法最终找到的解比次优基线更平坦38%，有助于获得更宽的极小值，从而提升泛化性能。
原文摘要 · Abstract (English)
Effective learning rate (LR) scheduling is crucial for training deep neural networks. However, popular pre-defined and adaptive schedulers can still lead to suboptimal generalization. This paper introduces VolSched, a novel adaptive LR scheduler inspired by the concept of volatility in stochastic processes like Geometric Brownian Motion to dynamically adjust the learning rate. By calculating the ratio between long-term and short-term accuracy volatility, VolSched increases the LR to escape plateaus and decreases it to stabilize training, allowing the model to explore the loss landscape more effectively. We evaluate VolSched on the CIFAR-100 dataset against a strong baseline using a standard augmentation pipeline. When paired with ResNet-18 and ResNet-34, our scheduler delivers consistent performance gains, improving top-1 accuracy by 1.4 and 1.3 percentage points respectively. Analysis of the loss curves reveals that VolSched promotes a longer exploration phase. A quantitative analysis of the Hessian shows that VolSched finds a final solution that is 38% flatter than the next-best baseline, allowing the model to obtain wider minima and hence better generalization performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。