针对大模型工具推理中的错误定位难题，提出精准识别关键错误步骤的方法。
Learning from the Irrecoverable: Error-Localized Policy Optimization for Tool-Integrated LLM Reasoning
- 通过二分搜索滚动树定位首个不可逆错误步骤
- 在数学、科学问答和代码执行任务中显著提升成功率与效率
- 适合需要高可靠性推理的智能代理系统研究者
工具集成推理（TIR）使大语言模型代理可通过规划、工具调用和迭代修正完成任务，但仅基于结果的强化学习面临奖励稀疏、延迟严重及步骤级信用分配薄弱的问题。在长时序TIR轨迹中，早期不可逆错误可能决定成败，因此精准定位首个不可逆步骤至关重要。本文提出误差局部化策略优化（ELPO），在固定采样预算下，通过二分搜索滚动树定位首个不可逆步骤；将所得树结构转化为稳定的学习信号，采用层次化优势归因；并应用误差局部自适应裁剪，增强对关键步骤及其后缀的修正更新。在数学、科学问答与代码执行等TIR基准上，ELPO在相同采样预算下持续超越强基线方法，在Pass@K与Major@K指标提升、滚动排名质量改善及工具调用效率方面均表现优异。代码将于近期公开。
原文摘要 · Abstract (English)
Tool-integrated reasoning (TIR) enables LLM agents to solve tasks through planning, tool use, and iterative revision, but outcome-only reinforcement learning in this setting suffers from sparse, delayed rewards and weak step-level credit assignment. In long-horizon TIR trajectories, an early irrecoverable mistake can determine success or failure, making it crucial to localize the first irrecoverable step and leverage it for fine-grained credit assignment. We propose Error-Localized Policy Optimization (ELPO), which localizes the first irrecoverable step via binary-search rollout trees under a fixed rollout budget, converts the resulting tree into stable learning signals through hierarchical advantage attribution, and applies error-localized adaptive clipping to strengthen corrective updates on the critical step and its suffix. Across TIR benchmarks in math, science QA, and code execution, ELPO consistently outperforms strong Agentic RL baselines under comparable sampling budgets, with additional gains in Pass@K and Major@K scaling, rollout ranking quality, and tool-call efficiency. Our code will be publicly released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。