arXiv:2607.28225cs.CV2026-07

让AI工具使用更真实,通过多智能体自检机制提升视觉推理可信度。

FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Verification

论文配图:FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Verification
图 1 · 摘自论文原文
  • 用VLM自动判断每步工具操作是否有助于答题,生成反馈信号
  • 在多个基准上准确率领先,工具使用真实性显著提升
  • 无需外部模型,可直接部署于实际推理场景

代理式视觉语言模型(VLM)通过穿插文本推理与具体工具调用(如裁剪、基于代码的图像处理),成为可靠且可解释的多模态推理新范式。然而,现有研究发现此类模型常不忠实使用工具:许多过程图像与问题无关(如裁剪错误区域或遗漏目标),但工具调用仍获全分,模型仍能正确回答。这类装饰性或错位的工具调用浪费计算资源,暴露模型依赖先验知识或原始图像而非检索到的证据。这源于两大局限:工具奖励无法区分有效与无效调用,工具反馈无有用性信号。为此,我们提出FaithEyes,一种多智能体自评判框架。具体而言,使用VLM判断每张过程图像是否有助于回答问题,并将判断结果注入推理上下文作为工具观测的一部分,同时以有用工具比例缩放工具奖励,抑制奖励劫持。为确保评估时判断可用,保持训练-测试一致性,我们设计多智能体架构,由模型自身作为子智能体,对主智能体的工具调用进行判断,避免推理时依赖外部模型。在适配开源数据上采用两阶段SFT+RL训练,FaithEyes在视觉感知与推理基准上达到竞争力或更优准确率,同时显著提升工具使用的真实性。主页见:https://github.com/Mosi-AI/FaithEyes。

原文摘要 · Abstract (English)

Agentic vision-language models (VLMs), which interleave textual reasoning with explicit tool calls such as cropping and code-based image manipulation, have emerged as a compelling paradigm for reliable and interpretable multimodal reasoning. However, recent studies have revealed that such models often use tools unfaithfully. Many process images are irrelevant to the question (e.g., the tool crops the wrong region or misses the queried target), yet the call still receives full credit and the model still answers correctly. Such decorative or misaligned tool calls waste computation and reveal that the model leans on prior knowledge or the original image rather than the evidence it retrieves. This may stem from two limitations of prevailing methods: the tool reward fails to distinguish useful from useless calls, and tool feedback carries no signal of usefulness. To this end, we introduce FaithEyes, a multi-agent self-judging framework. Concretely, we use a VLM to judge whether each process image helps answer the question. The judgement is injected into the reasoning context as part of the tool observation to help subsequent reasoning, and meanwhile is used to scale the tool reward by the helpful-tool ratio to suppress reward hacking. To keep judgement available at evaluation and thus ensure train-test consistency, we further design a multi-agent framework where the model itself serves as a subagent to judge the tool calls from main agent, eliminating any dependence on an external model at inference. Training via a two-stage SFT + RL pipeline on adapted open-source data, FaithEyes attains competitive or superior accuracy across visual perception and reasoning benchmarks, while markedly improving tool faithfulness. The homepage is at https://github.com/Mosi-AI/FaithEyes.

多智能体工具使用视觉推理自检机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。