让大模型智能重试错误步骤,减少无效交互。
Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry

- 通过定位关键错误点,只在出错处局部重试
- 在4个任务上平均提升11.5%的通过率
- 适合需要高效学习的长序列决策场景
大型语言模型(LLM)代理在长时序交互任务中展现出强大的决策能力,但仍难以有效利用失败轨迹:完整重试成本过高,而经验检索又易稀释关键信号。为此,我们提出PivoARL,一种基于关键错误感知的自反馈重试框架,用于提升LLM代理的经验利用率。PivoARL通过结构化反思识别出关键错误转折点,并仅从该关键状态进行局部重试,从而复用正确前缀,减少冗余交互。从信息增益视角看,关键重试将有用经验信号集中在错误边界附近,缓解了无差别经验使用导致的信号稀释问题。基于此,我们设计了一种关键感知的信用分配机制,奖励正确前缀并隔离错误后缀,并通过隐式反思返回优化反思质量。我们在4个代理任务和7个基于搜索的问答基准上进行了系统评估。结果表明,PivoARL在所有任务上均显著提升Pass@2/3,相比MetaRL平均提升约11.5%。此外,得益于关键转折点带来的对比偏好信号,PivoARL在超过80%的任务上也持续提升Pass@1。在Minesweeper环境中,其性能优于GiGPO超过45%,且相比全重试方法平均减少约42%的交互轮次。代码已公开于https://github.com/yuki-younai/PivoARL。
原文摘要 · Abstract (English)
Large language model (LLM) agents have shown strong decision-making capabilities in long-horizon interactive tasks, yet they still struggle to effectively leverage failed trajectories: full retries incur high interaction costs, while experience retrieval tends to dilute critical experience signals. To address this, we propose PivoARL, a self-feedback retry framework for experience exploitation in LLM agents. PivoARL identifies the pivotal erroneous turn through structured reflection and performs local retry only from the corresponding pivotal state, thereby reusing the correct prefix and reducing redundant interactions. From an information-gain perspective, we further show that pivotal retry concentrates useful experience signals near the error boundary, mitigating the signal dilution caused by state-agnostic experience utilization. Based on this insight, we design a pivotal-aware credit assignment mechanism that rewards correct prefixes while isolating erroneous suffixes, and optimize reflection quality through implicit reflection returns. We conduct a systematic evaluation on 4 agent tasks and 7 search-based QA benchmarks. Results show that PivoARL achieves significant improvements on Pass@2/3 across all tasks, with an average gain of about 11.5\% over MetaRL. Moreover, benefiting from contrastive preference signals induced by pivotal turns, PivoARL also consistently improves Pass@1 on over 80\% of the tasks. On Minesweeper environment, PivoARL improves over GiGPO by more than 45\% and reduces interaction turns by about 42\% on average compared with full-retry methods. Code is available at https://github.com/yuki-younai/PivoARL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。