用能量守恒原理自适应调节学习率，让优化更稳更快。
An Energy-Based Self-Adaptive Learning Rate for Stochastic Gradient Descent: Enhancing Unconstrained Optimization with VAV method
- 引入辅助变量r，基于能量法动态调整学习率
- 大学习率下仍稳定，初期收敛速度显著提升
- 适合追求高效训练的深度学习研究者
优化学习率仍是机器学习中的关键挑战，对模型稳定性和收敛效率至关重要。本文提出向量辅助变量（VAV）算法，一种面向无约束优化问题的能量基自适应学习率方法。该方法引入辅助变量 $r$，在无需回溯的情况下实现高效能量逼近，同时满足无条件能量耗散律。实验表明，VAV 在较大学习率下表现更稳定，且训练初期收敛更快。对比分析显示，VAV 在多种任务中优于随机梯度下降（SGD）。论文还给出了能量耗散律的严格证明，并在合理假设下建立了算法的收敛性。此外，$r$ 在实践中可作为训练损失的实证下界，提供了一种新调度策略，进一步提升算法性能。
原文摘要 · Abstract (English)
Optimizing the learning rate remains a critical challenge in machine learning, essential for achieving model stability and efficient convergence. The Vector Auxiliary Variable (VAV) algorithm introduces a novel energy-based self-adjustable learning rate optimization method designed for unconstrained optimization problems. It incorporates an auxiliary variable $r$ to facilitate efficient energy approximation without backtracking while adhering to the unconditional energy dissipation law. Notably, VAV demonstrates superior stability with larger learning rates and achieves faster convergence in the early stage of the training process. Comparative analyses demonstrate that VAV outperforms Stochastic Gradient Descent (SGD) across various tasks. This paper also provides rigorous proof of the energy dissipation law and establishes the convergence of the algorithm under reasonable assumptions. Additionally, $r$ acts as an empirical lower bound of the training loss in practice, offering a novel scheduling approach that further enhances algorithm performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。