根据训练停滞自动提升学习率，加速收敛并找到更优解
When to restart? Exploring escalating restarts on convergence
- 检测训练停滞后线性提升学习率，动态重启优化过程
- 在多个数据集上提升准确率0.5%-4.5%，优于传统调度器
- 适合追求更好收敛效果的深度学习模型调优者
学习率调度在深度神经网络优化中至关重要，直接影响收敛速度、稳定性和泛化能力。现有调度策略如余弦退火、周期性学习率和预热重启虽有成效，但多依赖固定或周期性触发，忽视训练动态（如停滞或收敛行为）。本文提出一种简单有效的策略——随机梯度下降带渐进重启（SGD-ER），在检测到收敛时自适应提升学习率。该方法监测训练进展，当出现停滞时触发重启，并线性递增学习率，以逃离尖锐局部极小值，探索损失曲面更平坦区域。我们在CIFAR-10、CIFAR-100和TinyImageNet上，对ResNet-18/34/50、VGG-16和DenseNet-101等架构进行了评估。相比标准调度器，SGD-ER在测试精度上提升0.5%至4.5%，验证了基于收敛感知的渐进重启策略对寻找更优局部极小值的有效性。
原文摘要 · Abstract (English)
Learning rate scheduling plays a critical role in the optimization of deep neural networks, directly influencing convergence speed, stability, and generalization. While existing schedulers such as cosine annealing, cyclical learning rates, and warm restarts have shown promise, they often rely on fixed or periodic triggers that are agnostic to the training dynamics, such as stagnation or convergence behavior. In this work, we propose a simple yet effective strategy, which we call Stochastic Gradient Descent with Escalating Restarts (SGD-ER). It adaptively increases the learning rate upon convergence. Our method monitors training progress and triggers restarts when stagnation is detected, linearly escalating the learning rate to escape sharp local minima and explore flatter regions of the loss landscape. We evaluate SGD-ER across CIFAR-10, CIFAR-100, and TinyImageNet on a range of architectures including ResNet-18/34/50, VGG-16, and DenseNet-101. Compared to standard schedulers, SGD-ER improves test accuracy by 0.5-4.5%, demonstrating the benefit of convergence-aware escalating restarts for better local optima.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。