解决多模态智能体推理中思考与工具使用不匹配的问题
Agent Explorative Policy Optimization for Multimodal Agentic Reasoning
- 通过重采样错误工具调用并优化思考前缀来弥合思考-行动差距
- 在9个基准上平均提升1.8个百分点，8B模型性能超32B基线
- 适合需要复杂工具调用的多模态推理任务研究者
具备扩展推理能力的视觉语言模型在复杂问题上表现优异，但许多现实问题需依赖外部工具，仅靠内部推理难以解决。因此，智能体推理需交替进行两种行为：以自包含方式思考（默认），以及高方差的工具使用（辅助动作）。这种结构不对称称为‘思考-行动差距’。在标准强化学习方法如GRPO下，训练中出现两个诊断性症状：工具使用仅出现在约30%的轨迹中；当使用时，同一组内约40%的问题所有工具调用均错误，导致学习信号被抑制。为此提出AXPO（Agent eXplorative Policy Optimization）：对所有错误工具调用的子组，固定思考前缀，重新采样工具调用及其后续动作，并结合不确定性选择前缀。在九个多模态基准上，SFT+AXPO相比SFT+GRPO平均提升1.8个百分点（Pass@1和Pass@4），8B规模模型在该设置下，其Pass@4性能超过32B基线模型，参数量仅为后者的1/4。
原文摘要 · Abstract (English)
Vision-language models with extended reasoning succeed on complex problems, but many real-world problems require external tools that internal reasoning alone often cannot resolve. Agentic reasoning therefore interleaves two behaviors with a structural asymmetry: thinking (the self-contained default) and tool use (a high-variance auxiliary acting). We refer to this asymmetry as the Thinking-Acting Gap. Under standard RL recipes like GRPO, the gap manifests as two diagnostic symptoms during training: tool use is attempted on only ~30% of rollouts, and when attempted, the tool-using rollouts within a group are all-wrong on ~40% of questions, suppressing the learning signal at the tool calls that needed it. We propose AXPO (Agent eXplorative Policy Optimization): for each all-wrong tool-using subgroup, AXPO fixes the thinking prefix and resamples the tool call and its continuation, paired with uncertainty-based prefix selection. Across nine multimodal benchmarks and three scales of Qwen3-VL-Thinking, SFT+AXPO outperforms SFT+GRPO at average (+1.8pp Pass@1 and +1.8pp Pass@4 at 8B on average) and 8B with SFT+AXPO surpasses the 32B Base on Pass@4 with 4 times fewer parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。