提出新型梯度估计方法,提升元学习效率与精度
Binomial Gradient-Based Meta-Learning for Enhanced Meta-Gradient Estimation
- 用二项式展开替代传统截断反向传播,提升元梯度估计精度
- 理论证明误差可超指数衰减,在少量迭代下表现更优
- 适用于数据少的下游任务,尤其适合模型无关元学习场景
元学习通过利用相关任务中的任务无关先验,实现对下游任务的快速适应,即使数据有限。基于梯度的元学习(GBML)依赖梯度下降(GD)将先验适配至新任务,但计算开销随梯度步数线性增长。现有方法采用截断反向传播近似元梯度,存在较大误差。本文提出二项式梯度元学习(BinomGBML),利用截断二项式展开进行元梯度估计,可在高效并行计算中引入更多信息。作为通用模型元学习(MAML)的实现,所提出的BinomMAML在温和条件下具有可证明的误差界,不仅优于现有方法,且误差呈超指数衰减。数值实验验证了理论分析,显示性能显著提升,仅略增计算开销。
原文摘要 · Abstract (English)
Meta-learning offers a principled framework leveraging \emph{task-invariant} priors from related tasks, with which \emph{task-specific} models can be fine-tuned on downstream tasks, even with limited data records. Gradient-based meta-learning (GBML) relies on gradient descent (GD) to adapt the prior to a new task. Albeit effective, these methods incur high computational overhead that scales linearly with the number of GD steps. To enhance efficiency and scalability, existing methods approximate the gradient of prior parameters (meta-gradient) via truncated backpropagation, yet suffer large approximation errors. Targeting accurate approximation, this work puts forth binomial GBML (BinomGBML), which relies on a truncated binomial expansion for meta-gradient estimation. This novel expansion endows more information in the meta-gradient estimation via efficient parallel computation. As a running paradigm applied to model-agnostic meta-learning (MAML), the resultant BinomMAML provably enjoys error bounds that not only improve upon existing approaches, but also decay super-exponentially under mild conditions. Numerical tests corroborate the theoretical analysis and showcase boosted performance with slightly increased computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。