提出新元学习算法FO-B-MAML,理论保证收敛且内存高效。
A New First-Order Meta-Learning Algorithm with Convergence Guarantees
- 从双层优化出发,用扰动问题解的导数定义元梯度
- 对称估计器偏差率降至O(δ²/³),优于现有方法
- 适合需要快速训练、低内存的深度模型元学习场景
利用先前经验学习新任务是智能系统的核心能力。尽管模型无关元学习(MAML)是主流方法,但其需计算二阶元梯度,带来显著的计算与内存开销。我们提出一种新的一阶变体——FO-B-MAML,源自双层优化视角。该框架通过扰动优化问题解的导数定义元梯度,支持多种有限差分估计;本文提出并分析了前向与对称两种简单有效的估计器。不同于FO-MAML和Reptile等存在不可消除偏差的方法,我们证明FO-B-MAML可收敛至元目标的驻点。值得注意的是,对称估计器达到改进的偏差率O(δ²/³),严格优于此前一阶理论。此外,我们发现MAML目标违反标准光滑性假设,其光滑常数随元梯度范数增长,这从理论上支持使用归一化或截断梯度方法(如SNGDM)而非原始梯度下降。实验验证表明,FO-B-MAML性能接近二阶MAML,且避免了二阶方法的“激活瓶颈”,在深层卷积网络与Transformer上仍保持恒定内存占用。
原文摘要 · Abstract (English)
Learning new tasks by leveraging prior experience is a fundamental trait of intelligent systems. While Model-Agnostic Meta-Learning (MAML) is a leading approach, it suffers from significant computational and memory overhead due to the requirement of computing second-order meta-gradients. We propose \textbf{FO-B-MAML}, a novel first-order variant of MAML derived from a bi-level optimization perspective. Our framework introduces a new expression of the meta-gradient, defined as the derivative of the solution of a perturbed optimization problem. This formulation allows the meta-gradient to be estimated using various finite difference methods; in this work, we propose and analyze two simple yet effective estimators: a forward and a symmetric approximation. Unlike existing first-order methods like FO-MAML and Reptile, which suffer from irreducible bias, we prove that FO-B-MAML converges to a stationary point of the meta-objective. Notably, the symmetric estimator achieves an improved $\mathcal{O}(δ^{2/3})$ bias rate, strictly enhancing previous first-order theory. Furthermore, we demonstrate that the MAML objective violates standard smoothness assumptions; we show instead that its smoothness constant grows with the norm of the meta-gradient. This property theoretically justifies the use of normalized or clipped-gradient methods (SNGDM) over vanilla gradient descent. Our empirical results validate these advancements: FO-B-MAML achieves high accuracy, closely following second-order MAML performance. Crucially, our method bypasses the ``activation bottleneck'' of second-order approaches, maintaining a flat memory footprint even when scaling to deep, activation-heavy CNNs and Transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。