让语言智能体学会从经验中提炼通用经验，提升未来新任务表现。
Training Language Agents to Learn from Experience
- 通过上下文训练框架，让智能体从交互轨迹中生成可复用的改进指令。
- 在ALFWorld和MiniHack上，训练后的智能体在未见过的任务族中表现显著提升。
- 适用于想研究自主学习与泛化能力的语言智能体开发者。
语言智能体可在交互环境中适应经验，但现有基于反思的方法仅能单次任务内自我修正。能否将经验提炼为可复用的教训以提升未来未见任务的表现尚不明确。本文提出上下文训练（ICT）任务，评估语言智能体跨任务自我提升能力。在ICT中，反思模型观察执行模型产生的轨迹，并生成旨在提升执行模型在后续未见任务中表现的系统提示。我们设计了一种基于强化学习的训练流程，直接从经验中学习反思策略，无需人工标注示例。在ALFWorld和MiniHack上的实验表明，训练后的反思模型在多数保留任务族上优于未经训练的基线，证明‘从经验中学习’的能力本身可被学习。部分情况下，模型还展现出超出训练基准的泛化能力，迁移到显著不同的环境。最后，我们发布MetaGym，一个用于构建元环境的通用Python库，支持未来对自提升语言智能体的研究。
原文摘要 · Abstract (English)
Language agents can adapt from experience in interactive environments, but current reflection-based methods can only self-correct within a single task instance. Whether such experience can be distilled into reusable lessons that improve performance on future unseen tasks remains unclear. We address this problem by introducing the In-context Training (ICT) task, a framework for evaluating cross-task self-improvement in language agents. In ICT, a reflector model observes trajectories collected by an actor model and generates system prompts intended to improve the actor's performance on future unseen tasks. We then propose an RL-based training pipeline for learning such reflections directly from experience, without human-provided examples. Across ALFWorld and MiniHack, our trained reflectors outperform an untrained baseline on most held-out task families, showing that the ability to learn from experience can itself be learned. In some cases, we observe generalisation beyond the benchmark on which the reflector was trained, to substantially different environments. Finally, we introduce MetaGym, a generic Python library for constructing meta-environments, enabling future research on self-improving language agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。