用元强化学习让语言模型主动探索环境，提升试错适应能力。
Meta-RL Induces Exploration in Language Agents
- 通过跨回合训练鼓励探索，优化长期奖励
- 测试时仅凭反馈反思即可调整策略，无需梯度更新
- 在3个任务上分别提升11%~19%，泛化能力更强
强化学习（RL）已用于训练大语言模型（LLM）代理与环境交互并解决多轮长周期任务。然而，这些代理在需要主动探索的任务中表现不佳，难以从试错中高效学习。本文提出LaMer，一种通用的元强化学习框架，使LLM代理能在测试时主动探索并利用环境反馈学习。LaMer包含两个关键组件：(i) 跨回合训练框架，以促进探索和长期奖励优化；(ii) 基于上下文的策略自适应，通过反思实现无需梯度更新的策略调整。在多个环境中的实验表明，相比传统RL基线，LaMer在Sokoban、MineSweeper和Webshop上分别取得11%、14%和19%的性能提升。此外，其在更复杂或未见任务上的泛化能力也优于传统RL训练代理。结果表明，元强化学习为语言代理引入探索提供了原则性方法，使其能通过学习到的探索策略更鲁棒地适应新环境。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has enabled the training of large language model (LLM) agents to interact with the environment and to solve multi-turn long-horizon tasks. However, the RL-trained agents often struggle in tasks that require active exploration and fail to efficiently adapt from trial-and-error experiences. In this paper, we present LaMer, a general Meta-RL framework that enables LLM agents to actively explore and learn from the environment feedback at test time. LaMer consists of two key components: (i) a cross-episode training framework to encourage exploration and long-term rewards optimization; and (ii) in-context policy adaptation via reflection, allowing the agent to adapt their policy from task feedback signal without gradient update. Experiments across diverse environments show that LaMer significantly improves performance over RL baselines, with 11%, 14%, and 19% performance gains on Sokoban, MineSweeper and Webshop, respectively. Moreover, LaMer also demonstrates better generalization to more challenging or previously unseen tasks compared to the RL-trained agents. Overall, our results demonstrate that Meta-RL provides a principled approach to induce exploration in language agents, enabling more robust adaptation to novel environments through learned exploration strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。