让大模型从错误中学习,提升长期任务的纠错能力。
Internalizing Agency from Reflective Experience
- 通过回溯反思环境反馈,生成可操作的经验。
- 在固定交互预算下,Pass@128 提升最高达14%。
- 适合需要持续纠错的复杂自主任务场景。
大语言模型正被广泛用作需长期互动、自我规划与纠错的自主代理。然而,现有以结果为导向的后训练方法(如基于可验证奖励的强化学习)主要优化最终成功信号,忽视了环境中丰富的反馈信息。这导致策略趋向分布锐化:仅重复少数已成功的行为,无法提升基于反馈的纠错能力,限制了长时任务中的问题求解能力(如 Pass@k)。为此,我们提出 LEAFE(从反思经验中学习反馈驱动的自主性),框架在探索阶段将环境反馈提炼为可行动的经验,回溯至早期决策点,尝试修正后的替代路径,并通过监督微调将这些经验指导的修正注入模型,从而增强未来交互中的纠错能力。在固定交互预算下,多个交互式编程与代理任务中,LEAFE 均显著优于基线模型,其 Pass@1 高于基础模型,且在 Pass@k 上超越基于结果驱动的基线(GRPO)和基于经验的方法(Early Experience),最高实现 14% 的性能提升。
原文摘要 · Abstract (English)
Large language models are increasingly deployed as autonomous agents that must plan, act, and recover from mistakes through long-horizon interaction with environments that provide rich feedback. However, prevailing outcome-driven post-training methods (e.g., RL with verifiable rewards) primarily optimize final success signals, leaving rich environment feedback underutilized. Consequently, they often lead to distribution sharpening: the policy becomes better at reproducing a narrow set of already-successful behaviors, while failing to improve the feedback-grounded agency needed to expand problem-solving capacity (e.g., Pass@k) in long-horizon settings. To address this, we propose LEAFE (Learning Feedback-Grounded Agency from Reflective Experience), a framework that internalizes recovery agency from reflective experience. Specifically, during exploration, the agent summarizes environment feedback into actionable experience, backtracks to earlier decision points, and explores alternative branches with revised actions. We then distill these experience-guided corrections into the model through supervised fine-tuning, enabling the policy to recover more effectively in future interactions. Across a diverse set of interactive coding and agentic tasks under fixed interaction budgets, LEAFE consistently improves Pass@1 over the base model and achieves higher Pass@k than outcome-driven baselines (GRPO) and experience-based methods such as Early Experience, with gains of up to 14% on Pass@128.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。