arXiv:2607.18830cs.LGcs.AI2026-07中稿 · the International …

用语言指令替代轨迹更新,让元强化学习更快更高效。

From Trajectories to Instructions: Language-Conditioned Meta-Reinforcement Learning

论文配图:From Trajectories to Instructions: Language-Conditioned Meta-Reinforcement Learning
图 1 · 摘自论文原文
  • 用任务指令的嵌入向量一步调整策略,取代传统梯度更新
  • 在BabyAI上达到同等或更好性能,训练速度提升显著
  • 适合有自然语言描述的任务,尤其适合快速适配新场景

模型无关元学习(MAML)是一种广泛使用的强化学习框架,通过学习可快速适应新任务的全局策略参数实现高效迁移。其训练包含内外两层循环:内层将全局参数适应为特定任务参数,外层评估并反向传播损失以优化全局参数。传统方法在内层依赖从环境收集轨迹并基于经验回报进行梯度更新,成本较高。我们观察到真正驱动全局参数学习的是外层,因此内层适应机制不必局限于梯度方式。由此提出LA-MAML(语言适配MAML),在内层直接利用任务语言指令的嵌入向量一步调整全局策略,取代轨迹收集与梯度更新。在BabyAI基准上的实验表明,LA-MAML在性能上与基线相当或更优,且每轮训练耗时显著降低。结果证明语言指令是轨迹更新的有效且高效的替代方案。

原文摘要 · Abstract (English)

Model-Agnostic Meta-Learning (MAML) is a widely used framework for reinforcement learning (RL) that enables efficient transfer by learning global policy parameters that can be rapidly adapted to new tasks. MAML training proceeds in two loops: an inner loop where the global parameters are adapted to task-specific parameters, and an outer loop where these task-specific parameters are evaluated and losses are back-propagated to improve the global parameters. Traditionally, the inner loop adaptation is performed by collecting trajectories from the task environment and applying gradient updates on the empirical expected return, which can be a costly operation. We note that it is the outer loop that drives the actual learning of global parameters, and therefore the inner loop adaptation mechanism need not be restricted to be gradient-based. This observation leads us to ask: Can we replace the inner loop trajectory collection and gradient update with a simpler, task-specific signal? In many practical settings, tasks are naturally accompanied by language instructions. Leveraging these instructions as a direct task-specific signal, we propose LA-MAML (Language Adapted MAML), which modifies the inner loop by adapting the global policy parameters in a single step through a learned embedding of the task instruction, replacing the inner loop trajectory collection and gradient-based updates. Experiments on the BabyAI benchmark demonstrate that LA-MAML achieves competitive or improved performance compared to baselines at a significantly lower per-iteration wall-clock training time. These results demonstrate that language instructions are an effective and efficient substitute for trajectory-based inner loop adaptation in meta RL.

元强化学习语言指令快速适应

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。