让大模型学会从工具调用错误中自我修复,提升多轮任务成功率。
Robust Tool Use via Fission-GRPO: Learning to Recover from Execution Errors
- 在强化学习中动态生成纠错训练样本,利用实时反馈指导恢复。
- 在BFCL v4上使Qwen3-8B的错误恢复率提升5.7%,总准确率提高4.0%。
- 适用于多种复杂任务场景,尤其适合需要强容错能力的智能体系统。
大型语言模型虽能有效调用工具,但在多轮执行中仍易出错:一旦工具调用失败,小型模型常陷入重复无效重试,无法根据反馈进行恢复。此问题源于当前训练范式未显式教授模型如何纠错。标准强化学习将丰富的失败经验压缩为稀疏负奖励,而预先收集的纠错数据集又与策略演化后的失败模式不匹配。为此,我们提出Fission-GRPO框架,将执行错误转化为强化学习训练循环内的在线纠错监督。核心机制是通过微调的错误模拟器为每条失败轨迹注入诊断反馈,并生成多个新的恢复回滚路径,从而让模型基于自身探索中实际犯的错误进行学习,而非依赖静态的预设错误案例。在BFCL v4 Multi-Turn测试中,该方法使Qwen3-8B的错误恢复率提升5.7个百分点,整体准确率从42.75%增至46.75%,优于所有强化学习基线及专用工具调用代理。该方法在TAU-Bench和TAU2-Bench上也取得领先表现,部分设置下性能提升高达+17.4%。
原文摘要 · Abstract (English)
Large language models (LLMs) can call tools effectively, yet they remain brittle in multi-turn execution: after a tool-call error, smaller models often fall into repetitive invalid re-invocations instead of interpreting the feedback and recovering. This failure mode persists because current training paradigms do not explicitly teach models how to recover from execution errors. In particular, standard reinforcement learning (RL) collapses rich failure experience into sparse negative rewards, while pre-collected error-correction datasets become mismatched to the policy's evolving failure modes. To bridge this gap, we propose Fission-GRPO, a framework that converts execution errors into on-policy corrective supervision within the RL training loop. Our core mechanism fissions each failed trajectory into a new training instance by augmenting it with diagnostic feedback from a fine-tuned Error Simulator, then resampling multiple recovery rollouts on-policy. This enables the model to learn from the precise errors it makes during exploration, rather than from static, pre-collected error cases. On BFCL v4 Multi-Turn, Fission-GRPO improves the error recovery rate of Qwen3-8B by 5.7% absolute and overall accuracy by 4.0% (from 42.75% to 46.75%), outperforming both RL baselines and specialized tool-use agents. The method further generalizes to TAU-Bench and TAU2-Bench, achieving leading results across most settings with gains up to +17.4%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。