根据损失变化动态调整学习率，加速模型收敛。
Dynamic Learning Rate Scheduling based on Loss Changes Leads to Faster Convergence
- 依据当前损失值自适应调整学习率，无需预设衰减策略。
- 在70亿参数模型上，训练速度更快且收敛更稳定。
- 理论证明收敛性，适合追求高效训练的从业者使用。
尽管优化器研究取得显著进展，多数工作仍采用余弦或指数衰减等常见调度策略。本文提出一种新型调度器GreedyLR，根据当前损失值动态调整学习率。我们在多个NLP、CV及大语言模型任务上进行实验，涵盖高达70亿参数的微调与预训练场景。结果表明，该方法在准确率、速度和收敛性方面均优于多种前沿调度器。我们还提供了对GreedyLR算法的理论分析，包括收敛性证明及最大化收敛速率的最优缩放因子$F$的推导，并通过实验证明其在真实噪声环境下的鲁棒性。该调度器实现简单、计算高效，可作为训练默认调度方案。
原文摘要 · Abstract (English)
Despite significant advances in optimizers for training, most research works use common scheduler choices like Cosine or exponential decay. In this paper, we study \emph{GreedyLR}, a novel scheduler that adaptively adjusts the learning rate during training based on the current loss. To validate the effectiveness of our proposed scheduler, we conduct experiments on several NLP, CV, and LLM tasks with up to $7B$ parameters, including both fine-tuning and pre-training experiments. The results show that our approach outperforms several state-of-the-art schedulers in terms of accuracy, speed, and convergence. We also provide a theoretical analysis of the GreedyLR algorithm, including a proof of convergence and derivation of the optimal scaling factor $F$ that maximizes the convergence rate, along with experiments to show robustness of the algorithm to realistic noisy landscapes. Our scheduler is easy to implement, computationally efficient, and could be considered a good default scheduler for training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。