用预训练模型加速Adam优化器调参，提升效率与泛化能力。
Tune My Adam, Please!
- 基于任务集预训练的代理模型，结合曲线增强方法提升调参效率
- 在任务集上实现更准确的学习曲线外推，加速超参数优化
- 对分布外任务表现优异，适合需要快速调参的场景
Adam优化器在深度学习中仍被广泛使用，有效调参对性能至关重要，但过程繁琐且成本高。冻结-解冻贝叶斯优化（Freeze-thaw BO）是一种低成本超参数调优的新兴方法，但受限于缺乏对超参数影响学习过程的先验知识的通用代理模型。本文提出Adam-PFN，一种针对Adam超参数的新型代理模型，其在TaskSet上的学习曲线数据上预训练，并结合一种新提出的曲线增强方法CDF-augment，可人工增加可用训练样本数量。该方法显著提升了学习曲线外推效果，加速了在TaskSet评估任务上的超参数优化，且在分布外（OOD）任务上表现强劲。
原文摘要 · Abstract (English)
The Adam optimizer remains one of the most widely used optimizers in deep learning, and effectively tuning its hyperparameters is key to optimizing performance. However, tuning can be tedious and costly. Freeze-thaw Bayesian Optimization (BO) is a recent promising approach for low-budget hyperparameter tuning, but is limited by generic surrogates without prior knowledge of how hyperparameters affect learning. We propose Adam-PFN, a new surrogate model for Freeze-thaw BO of Adam's hyperparameters, pre-trained on learning curves from TaskSet, together with a new learning curve augmentation method, CDF-augment, which artificially increases the number of available training examples. Our approach improves both learning curve extrapolation and accelerates hyperparameter optimization on TaskSet evaluation tasks, with strong performance on out-of-distribution (OOD) tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。