提出高效长时程学习算法，让神经优化器更省力、更强大。
Efficient Long-Horizon Learning for Learned Optimization
- 通过重分配计算资源到更长失败阶段，实现高效长时程元训练。
- 在语言和图像任务上显著提升优化器长期性能与泛化能力。
- 元训练耗时低于7个H100 GPU小时，适合快速实验与部署。
学习型优化旨在通过元学习在任务分布上训练小型神经网络优化器，以超越手工设计的优化器（如Adam和Muon）。尽管近期研究在学习型优化器（LOs）的架构设计和归纳偏置方面取得进展，但其元训练仍偏向于特定任务上的短时程学习，导致计算冗余，使LOs难以与手工优化器竞争。本文提出高效长时程（ELO）学习，一种高效的元训练算法：(1) 将浪费的元训练计算重新分配至更长的失败阶段，实现高效的长时程学习；(2) 引入解耦渐进式专家监督，提供稳定元学习信号，进一步提升泛化能力。实证研究评估了ELO在元素级与矩阵级LOs上的元训练效果。在下游语言建模（GPT-2-124M/350M on FineWeb）和图像分类（ViT-B/16, ResNet-50 on ImageNet-1K）任务中，ELO显著提升基础LOs的长时程性能与跨分布泛化能力。特别地，ELO-Celo2在所有测试任务上均优于调优后的AdamW，且在语言建模上保持与Muon相当的竞争力。值得注意的是，所有ELO基线元训练耗时均低于7个H100 GPU小时。
原文摘要 · Abstract (English)
Learned optimization aims to improve upon hand-designed optimizers (e.g., Adam and Muon) by meta-learning small neural network optimizers over a distribution of tasks. While recent work has greatly advanced the architectural design and inductive biases of learned optimizers (LOs), their meta-training remains biased toward short-unroll learning on particular tasks, resulting in redundant computation and leaving LOs often unable to compete with hand-designed optimizers. We introduce Efficient Long-hOrizon (ELO) learning, an efficient meta-training algorithm that (1) reallocates wasted meta-training compute to longer failure regimes, achieving efficient long-horizon learning, and (2) enforces decoupled progressive expert supervision, providing stable meta-learning signals that additionally improve the generalization of LOs. Our empirical study evaluates ELO for meta-training both element-wise and matrix-based LOs. Across downstream language modeling (GPT-2-124M/350M on FineWeb) and image classification (ViT-B/16, ResNet-50 on ImageNet-1K) tasks, ELO substantially improves the long-unroll performance and out-of-distribution generalization of the base LOs. In particular, ELO-Celo2 consistently outperforms well-tuned AdamW across all evaluated tasks, while remaining competitive with Muon on language modeling. \textit{Notably, all ELO baselines require less than 7 H100 GPU-hours for meta-training.}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。