arXiv:2510.00212cs.LGcs.AI2025-10被引 1

提出任务导向近似方法,加速元强化学习收敛并降低计算开销

Directed-MAML: Meta Reinforcement Learning Algorithm with Task-directed Approximation

  • 引入任务导向的一阶近似,替代部分二阶梯度计算
  • 在多个环境上实现更快收敛与更低计算成本
  • 可通用集成到FOMAML、Meta-SGD等算法中

模型无关元学习(MAML)是一种适用于监督学习和强化学习的通用元学习框架。然而将其应用于元强化学习(meta-RL)面临显著挑战:一是依赖二阶梯度计算,带来巨大计算与内存开销;二是嵌套优化结构增加问题复杂性,更难收敛至全局最优。为此,本文提出定向MAML(Directed-MAML),一种新的任务导向型元强化学习算法。在执行二阶梯度步骤前,该算法通过额外的一阶任务导向近似,预估二阶梯度的影响,从而加速最优解收敛并降低计算成本。实验表明,在CartPole-v1、LunarLander-v2及双车交叉路口场景中,Directed-MAML优于基于MAML的基线模型,在计算效率和收敛速度上均有提升。此外,任务导向近似可有效融入FOMAML与Meta-SGD等其他元学习算法,进一步提高效率与速度。

原文摘要 · Abstract (English)

Model-Agnostic Meta-Learning (MAML) is a versatile meta-learning framework applicable to both supervised learning and reinforcement learning (RL). However, applying MAML to meta-reinforcement learning (meta-RL) presents notable challenges. First, MAML relies on second-order gradient computations, leading to significant computational and memory overhead. Second, the nested structure of optimization increases the problem's complexity, making convergence to a global optimum more challenging. To overcome these limitations, we propose Directed-MAML, a novel task-directed meta-RL algorithm. Before the second-order gradient step, Directed-MAML applies an additional first-order task-directed approximation to estimate the effect of second-order gradients, thereby accelerating convergence to the optimum and reducing computational cost. Experimental results demonstrate that Directed-MAML surpasses MAML-based baselines in computational efficiency and convergence speed in the scenarios of CartPole-v1, LunarLander-v2 and two-vehicle intersection crossing. Furthermore, we show that task-directed approximation can be effectively integrated into other meta-learning algorithms, such as First-Order Model-Agnostic Meta-Learning (FOMAML) and Meta Stochastic Gradient Descent(Meta-SGD), yielding improved computational efficiency and convergence speed.

元强化学习MAML加速收敛一阶近似

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。