用梯度步数评估模型,发现大学习率能更快训练出好模型。
Gradient-Descent Steps to Success over Mean Accuracy: A Paradigm Shift for ML

- 以训练所需梯度步数为评价标准,取代传统准确率优先。
- 大学习率可显著减少训练步数,提升效率并促进泛化。
- 适合关注训练成本、追求高效模型的工程师和研究者。
传统机器学习评估侧重于最高准确率,忽略计算成本。本文提出新范式:以达到目标准确率所需的总梯度下降步数作为计算努力的显式度量。基于此,我们扩展该指标至所有梯度下降训练的模型,并证明最小化该努力等价于一种新型自动机器学习(AutoML)。在11种不同模型和5个标准分类数据集上的实验揭示,最优超参数始终偏好异常大的学习率。关键发现是,大学习率带来的快速、激进的损失曲面遍历不仅促进泛化(如超收敛现象),还能统计上最小化预期计算努力。此外,我们识别出优化策略的相变:低准确率目标只需单次训练,而逼近模型性能极限则需大量独立短周期重启。最后,该努力驱动范式为模型选择提供稳健框架,使从业者可根据问题难度选择算法,或在固定梯度步数预算下最大化准确率。
原文摘要 · Abstract (English)
Traditional evaluation of machine learning (ML) models typically focuses on achieving the maximum possible accuracy irrespective of the computational cost. In this article, we propose a paradigm shift towards evaluating performance based on computational effort-explicitly defined here as the total number of gradient descent steps required to reach an acceptable level of accuracy with high probability. Building upon the concept of computational effort originally introduced by Koza for Genetic Programming, we extend this metric to any ML model trained via gradient descent. Furthermore, we demonstrate that minimising this effort acts as a novel form of Automatic Machine Learning (AutoML). By evaluating it across 11 diverse ML models and five standard classification datasets, we uncover significant insights into the dynamics of gradient-based learning. Our findings reveal that optimal hyper-parameters consistently favour unusually large learning rates. Crucially, we demonstrate that the rapid, aggressive landscape traversal enabled by these large rates not only promotes generalisation-as seen in phenomena like superconvergence-but also statistically minimises the expected computational effort for training. Furthermore, we identify distinct phase transitions in the optimal search strategy: while a single training run suffices for lower accuracy targets, reaching a model's performance limit requires a dramatic shift towards conducting numerous independent, short restarts. Finally, we illustrate how this effort-based paradigm provides a robust framework for model selection, allowing practitioners to choose optimal algorithms based on the difficulty of a problem as perceived by different models for a given target accuracy, or to maximise the achievable accuracy for a fixed budget of gradient descent steps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。