让AI看图推理时言行一致,避免说一套做一套。
Walk the Talk: Bridging the Reasoning-Action Gap for Thinking with Images via Multimodal Agentic Policy Optimization
- 要求模型为视觉工具结果生成文字描述,强制对齐思考与动作。
- 新奖励机制结合语义匹配与任务奖励,提升推理准确性。
- 在多个视觉推理数据集上表现更优,适合需要精准视觉决策的场景。
多模态大语言模型(MLLM)推动了模型在多轮推理中主动调用视觉工具以实现“看图思考”。然而,传统强化学习依赖结果奖励的做法忽略了文本合理性掩盖执行失败的问题:模型可能看似合理地推理,但实际执行的视觉操作却模糊或无关。这种思维与行为的不一致会随多轮推理累积噪声,严重削弱模型的多模态推理能力,甚至导致训练崩溃。本文提出多模态智能体策略优化(MAPO),弥合多模态思维链(MCoT)中文本推理与视觉动作之间的差距。具体而言,MAPO要求模型为工具获取的视觉内容生成明确的文字描述,并设计一种新型优势估计方法,将这些描述与实际观察的语义一致性与任务奖励耦合。理论分析表明,该方法能有效降低梯度方差。大量实验验证,该方法在多个视觉推理基准上均取得更优性能。
原文摘要 · Abstract (English)
Recent advancements in Multimodal Large Language Models (MLLMs) have incentivized models to ``think with images'' by actively invoking visual tools during multi-turn reasoning. The common Reinforcement Learning (RL) practice of relying on outcome-based rewards ignores the fact that textual plausibility often masks executive failure, meaning that models may exhibit intuitive textual reasoning while executing imprecise or irrelevant visual actions within their agentic reasoning trajectories. This reasoning-action discrepancy introduces noise that accumulates throughout the multi-turn reasoning process, severely degrading the model's multimodal reasoning capabilities and potentially leading to training collapse. In this paper, we introduce Multimodal Agentic Policy Optimization (MAPO), bridging the gap between textual reasoning and visual actions generated by models within their Multimodal Chain-of-Thought (MCoT). Specifically, MAPO mandates the model to generate explicit textual descriptions for the visual content obtained via tool usage. We then employ a novel advantage estimation that couples the semantic alignment between these descriptions and the actual observations with the task reward. Theoretical findings are provided to justify the rationale behind MAPO, which inherently reduces the variance of gradients, and extensive experiments demonstrate that our method achieves superior performance across multiple visual reasoning benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。