让工具调用智能体学会自我纠错，提升复杂任务成功率。
ReGRPO: Reflection-Augmented Policy Optimization for Tool-Using Agents
- 通过模拟失败场景生成反思数据，指导错误修复。
- 在相同模型和工具下，任务成功率超越现有开源基线。
- 适合需要可靠工具调用的多步复杂任务研究者。
工具增强型视觉语言模型可通过调用外部工具解决多模态、多步骤任务，但实际应用中仍易出错。现有方法存在两大缺陷：监督微调（SFT）仅基于成功轨迹，缺乏失败后恢复信号；稀疏的轨迹级强化学习奖励无法定位具体失败步骤及修复方式。本文提出ReGRPO（反思增强组相对策略优化），通过结构化反思数据引擎：执行近失败动作收集真实故障观测，构建包含错误类型、证据与修复计划的反思三元组（ErrorType, Evidence, FixPlan），并配对修正动作用于预训练。随后，在局部轨迹中联合优化反思标记与修正动作，采用组相对优势，并引入反思成本项以减少冗余反思。在GTA和GAIA数据集上的实验表明，使用相同骨干网络与工具套件时，ReGRPO持续优于强开源基线，且在对比的开源控制器中表现最佳。代码与反思数据已公开于https://github.com/showlab/ReGRPO。
原文摘要 · Abstract (English)
Tool-augmented vision-language models (VLMs) can solve multimodal, multi-step tasks by calling external tools, yet they remain fragile in practice. Existing works have two common gaps. Supervised fine-tuning (SFT) is built mostly on successful trajectories and offers little signal for recovery after tool failures, while sparse trajectory-level RL rewards provide limited guidance on which step failed and how to repair it. We introduce ReGRPO (Reflection-augmented Group Relative Policy Optimization), a framework that learns reflection-guided correction in tool-using agents. ReGRPO starts with a structured reflective data engine: we execute near-miss actions to collect grounded failure observations, then build Reflection-of-Thought triplets (ErrorType, Evidence, FixPlan) paired with corrected actions for warm-start SFT. We then optimize reflection tokens and corrective actions jointly within local trajectories using group-relative advantages, and include a reflection-cost term to reduce unnecessary reflection. Experiments on GTA and GAIA show that, under the same backbone and tool suite, ReGRPO consistently outperforms strong open-source baselines and achieves the best results among the compared open-source controllers. Code and RoT data are available at https://github.com/showlab/ReGRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。