新元学习算法无需历史参数,内存消耗显著降低且保证收敛。
Memory-Reduced Meta-Learning with Guaranteed Convergence
- 不依赖历史参数/梯度,直接计算上层梯度,减少内存开销。
- 上层优化迭代次数增加时,误差以次线性速度下降,批任务数越多越优。
- 理论证明可收敛至精确解,计算复杂度与现有方法相当,适合资源受限场景。
基于优化的元学习方法因能仅用少量数据快速适应新任务而日益受到关注。然而,现有方法如MAML、ANIL及其变体通常采用反向传播估计上层梯度,需保存历史底层参数/梯度,导致每轮迭代中计算与内存开销较大。本文提出一种新元学习算法,可避免使用历史参数/梯度,显著降低每轮迭代的内存成本。此外,我们证明该算法在上层优化迭代次数增加时,收敛误差以次线性速度下降;当采样任务批次增大时,误差同样以次线性速度衰减。在确定性元学习情形下,进一步证明其可收敛至精确解。算法计算复杂度为$/mathcal{O}(ε^{-1})$,与不使用历史参数的已有收敛结果一致。在多个元学习基准测试上的实验结果验证了该方法的有效性。
原文摘要 · Abstract (English)
The optimization-based meta-learning approach is gaining increased traction because of its unique ability to quickly adapt to a new task using only small amounts of data. However, existing optimization-based meta-learning approaches, such as MAML, ANIL and their variants, generally employ backpropagation for upper-level gradient estimation, which requires using historical lower-level parameters/gradients and thus increases computational and memory overhead in each iteration. In this paper, we propose a meta-learning algorithm that can avoid using historical parameters/gradients and significantly reduce memory costs in each iteration compared to existing optimization-based meta-learning approaches. In addition to memory reduction, we prove that our proposed algorithm converges sublinearly with the iteration number of upper-level optimization, and the convergence error decays sublinearly with the batch size of sampled tasks. In the specific case in terms of deterministic meta-learning, we also prove that our proposed algorithm converges to an exact solution. Moreover, we quantify that the computational complexity of the algorithm is on the order of $\mathcal{O}(ε^{-1})$, which matches existing convergence results on meta-learning even without using any historical parameters/gradients. Experimental results on meta-learning benchmarks confirm the efficacy of our proposed algorithm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。