arXiv:2511.19661cs.CV2025-11被引 22

让AI看图推理时真用工具,不靠猜答案。

CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimization

  • 用代码表示图像操作,通过强化学习精准奖励真实有用的操作。
  • 在视觉搜索任务中,正确率提升同时忠实使用工具的比例大幅提高。
  • 适合研究可信AI、多模态推理与工具调用系统的开发者参考。

代理型视觉语言模型常通过调用图像操作来实现‘用图像思考’,但研究表明,高最终答案准确率背后隐藏着不忠实的视觉推理:模型可能在无关区域调用工具或完全忽略工具输出,仍能猜对答案。本文提出一种忠实性评估协议,检测中间视觉工具输出(如裁剪区域)是否包含所查询证据。结果发现,现有视觉代理虽有高准确率,但忠实工具使用率很低。为此,我们提出CodeV,一种基于代码的视觉代理,采用工具感知策略优化(TAPO)训练。TAPO是一种过程级强化学习框架,通过直接作用于视觉工具输入输出的密集奖励,而非思维链标记,使监督更易验证且抗奖励作弊。CodeV将视觉工具表示为可执行Python代码,奖励基于问题与工具输出逐步计算,鼓励必要且证据一致的工具使用。在两阶段SFT+RL训练流程中,CodeV在相关视觉搜索基准上达到竞争力或更优准确率,同时显著提升忠实工具使用率。此外,在多种多模态推理与数学基准上表现优异,表明显式监督中间工具行为对构建可信自主视觉推理系统至关重要。

原文摘要 · Abstract (English)

Agentic vision-language models are increasingly trained to "think with images" by calling image operations. However, we show that high final-answer accuracy often hides unfaithful visual reasoning: models may invoke tools on irrelevant regions or ignore tool outputs entirely, yet still guess the correct answer. In this work, we first propose a faithfulness evaluation protocol that measures whether intermediate visual tool outputs (e.g., crops) actually contain the queried evidence. This reveals that recent visual agents achieve high final-answer accuracy but exhibit low rates of faithful tool-use on visual search benchmarks. We then introduce CodeV, a code-based visual agent trained with Tool-Aware Policy Optimization (TAPO). TAPO is a process-level RL framework that augments GRPO with dense rewards defined directly on visual tool inputs and outputs, rather than on chain-of-thought tokens, making supervision easier to verify and less susceptible to reward hacking. CodeV represents visual tools as executable Python code, and TAPO assigns step-wise rewards based solely on the question and tool output, encouraging both necessary and evidence-consistent tool use. In a two-stage SFT+RL pipeline, CodeV achieves competitive or superior accuracy while substantially increasing faithful tool-use rates on related visual search benchmarks. Beyond visual search, CodeV attains strong performance on a range of multimodal reasoning and math benchmarks, suggesting that explicitly supervising intermediate tool behavior is crucial for building trustworthy, agentic visual reasoning systems.

视觉推理工具调用强化学习可信AI

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。