用24小时训练出能跨任务通用的高效学习型优化器。
Celo: Training Versatile Learned Optimizers on a Compute Diet
- 设计轻量架构与训练流程，提升优化器元泛化能力。
- 仅用24小时训练即在新任务上超越主流优化器。
- 适合追求低资源、强泛化优化器的研究者使用。
学习型优化器作为手写优化器的替代方案，有望发现更优的更新规则，实现神经网络的快速、无需超参数调整训练。实用的学习型优化器需具备强元泛化能力——即在元训练后可直接应用于新任务。现有顶尖方法VeLO（Metz等，2022）需4000个TPU月的大量多样任务和巨大算力才能实现元泛化，难以进一步改进。本文识别出学习型优化器架构与元训练流程中的关键要素，提出可量化评估优化器性能的评测指标。所提出的Celo方法显著提升元泛化性能，在仅24个GPU小时的元训练下，仍优于调优后的主流优化器，在多种分布外任务上表现更佳。
原文摘要 · Abstract (English)
Learned optimization has emerged as a promising alternative to hand-crafted optimizers, with the potential to discover stronger learned update rules that enable faster, hyperparameter-free training of neural networks. A critical element for practically useful learned optimizers, that can be used off-the-shelf after meta-training, is strong meta-generalization: the ability to apply the optimizers to new tasks. Recent state-of-the-art work in learned optimizers, VeLO (Metz et al., 2022), requires a large number of highly diverse meta-training tasks along with massive computational resources, 4000 TPU months, to achieve meta-generalization. This makes further improvements to such learned optimizers impractical. In this work, we identify several key elements in learned optimizer architectures and meta-training procedures that can lead to strong meta-generalization. We also propose evaluation metrics to reliably assess quantitative performance of an optimizer at scale on a set of evaluation tasks. Our proposed approach, Celo, makes a significant leap in improving the meta-generalization performance of learned optimizers and also outperforms tuned state-of-the-art optimizers on a diverse set of out-of-distribution tasks, despite being meta-trained for just 24 GPU hours.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。