自动寻找最优学习率，无需人工调参或预设初始值
ExpTest: Automating Learning Rate Searching and Tuning with Insights from Linearized Neural Networks
- 基于线性化神经网络的损失曲线进行实时假设检验
- 在多个模型和任务上达到当前最优性能，无需学习率调度
- 计算开销小，对超参数不敏感，适合快速部署
深度神经网络训练中的超参数调优仍是重大挑战，常需手动或耗时的网格搜索，增加资源成本并阻碍机器学习普及。初始学习率的选择尤为关键。现有自动化方法仍需人工设定初始学习率，而无需此步骤的方法则性能较差。本文提出ExpTest，一种用于深度神经网络训练中初始学习率搜索与后续调优的高效方法。该方法基于线性化神经网络的洞察与损失曲线形态，将其视为实时信号进行假设检验。我们从数学上证明了ExpTest的有效性，并提供了实证支持。ExpTest仅需极少额外开销，对超参数选择具有鲁棒性，在多种任务与架构上均实现当前最优表现，且无需初始学习率设定或学习率调度。
原文摘要 · Abstract (English)
Hyperparameter tuning remains a significant challenge for the training of deep neural networks (DNNs), requiring manual and/or time-intensive grid searches, increasing resource costs and presenting a barrier to the democratization of machine learning. The global initial learning rate for DNN training is particularly important. Several techniques have been proposed for automated learning rate tuning during training; however, they still require manual searching for the global initial learning rate. Though methods exist that do not require this initial selection, they suffer from poor performance. Here, we present ExpTest, a sophisticated method for initial learning rate searching and subsequent learning rate tuning for the training of DNNs. ExpTest draws on insights from linearized neural networks and the form of the loss curve, which we treat as a real-time signal upon which we perform hypothesis testing. We mathematically justify ExpTest and provide empirical support. ExpTest requires minimal overhead, is robust to hyperparameter choice, and achieves state-of-the-art performance on a variety of tasks and architectures, without initial learning rate selection or learning rate scheduling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。