让大模型智能体学会事后反思任务进展，提升长任务完成率
Retrospective Progress-Aware Self-Refinement for LLM Agent Training
- 先执行再回溯：执行完任务后，用最终结果反推每一步进展
- 事后反思比实时提示更有效，可使成功率提升12%
- 无需持续人工标注，适合训练复杂长任务智能体
基于大模型的智能体在强化学习训练中仅优化单步动作预测，缺乏对任务进展的元认知意识，导致长时序任务扩展受限。初步研究发现，实时进度提示会损害性能，而事后示范有效，但这种能力无法仅通过结果奖励训练获得。本文提出RePro（Retrospective Progress-Aware Training）框架，采用先执行后反思的滚动范式：智能体在线执行动作后，基于完整轨迹和已知结果，回溯评估各步骤进展。RePro初始阶段通过少量外部示范进行反思格式预热，随后通过复合奖励机制的RePro-PO进一步训练，实现无需持续外部监督的自生成进展信号。在WebShop、ALFWorld和Sokoban上的实验表明，RePro显著提升Qwen系列模型性能，最高实现12%的绝对成功率提升。
原文摘要 · Abstract (English)
LLM-based agents trained with reinforcement learning optimize step-wise action prediction but lack metacognitive awareness of task progress, inducing a gap that hinders long-horizon scaling. A pilot study reveals that online progress prompting hurts performance while retrospective demonstrations help, yet this capability cannot emerge from outcome-reward training alone. We present RePro, Retrospective Progress-Aware Training, a framework that trains agents to self-generate progress signals via a forward-then-reflect rollout paradigm: the agent executes actions online, then retrospectively reassesses its step-wise progress given the completed trajectory and known outcome. RePro initializes with a Retrospection Warmup that teaches reflection format from minimal external demonstrations, then further trains through RePro-PO with a composite reward that produces self-generated signals without continuous external supervision. Experiments on WebShop, ALFWorld, and Sokoban show that RePro enhances the Qwen family's performance, with up to $12\%$ absolute success rate gains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。