用学习方法动态调整分数阶梯度下降的超参数,提升优化性能。
Enhancing Fractional Gradient Descent with Learned Optimizers
- 通过元学习自动调整分数阶梯度下降的超参数
- 在多个任务中表现优于静态超参数设置,接近黑盒元优化器
- 适合研究分数阶优化机制及超参数调度的学者
分数阶梯度下降(FGD)通过引入分数阶微积分为机器学习优化提供了新路径,虽已展现潜力,但在收敛性与超参数选择方面仍面临挑战,尤其在神经网络等非凸场景下难以有效调度。为此,我们提出学习优化Caputo分数阶梯度下降(L2O-CFGD),通过元学习动态调整CFGD的超参数。其元学习调度策略在多个任务中超越了经过广泛搜索得到的静态超参数配置,并在部分任务中达到与全黑箱元优化器相当的性能。L2O-CFGD可作为研究者识别高性能超参数、理解分数微分历史依赖性的有力工具。
原文摘要 · Abstract (English)
Fractional Gradient Descent (FGD) offers a novel and promising way to accelerate optimization by incorporating fractional calculus into machine learning. Although FGD has shown encouraging initial results across various optimization tasks, it faces significant challenges with convergence behavior and hyperparameter selection. Moreover, the impact of its hyperparameters is not fully understood, and scheduling them is particularly difficult in non-convex settings such as neural network training. To address these issues, we propose a novel approach called Learning to Optimize Caputo Fractional Gradient Descent (L2O-CFGD), which meta-learns how to dynamically tune the hyperparameters of Caputo FGD (CFGD). Our method's meta-learned schedule outperforms CFGD with static hyperparameters found through an extensive search and, in some tasks, achieves performance comparable to a fully black-box meta-learned optimizer. L2O-CFGD can thus serve as a powerful tool for researchers to identify high-performing hyperparameters and gain insights on how to leverage the history-dependence of the fractional differential in optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。