arXiv:2608.06270cs.AI2026-08被引 1

发现视觉工具使用大多只是幻觉,实际并未真正提升模型推理能力。

The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images

论文配图:The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images
图 1 · 摘自论文原文
  • 构建因果图分离观察路径与操作捷径,逐层干预验证视觉证据作用
  • 6个模型5个基准测试中,仅少数情况有真实效果,多数为无效操作
  • 揭示两种失败模式:看得见却没用,或看了但顺序混乱

“用图像思考”范式为多模态大模型赋予裁剪、缩放等视觉操作能力。然而,使用这些操作的模型在消耗更高token成本的情况下,性能提升微弱甚至下降,且频繁裁剪无关区域,对直接推理能答对的问题反而出错。我们探究返回的视觉证据是否对答案具有因果影响。通过将视觉工具使用建模为因果图,区分观测中介路径与动作诱导捷径,我们在策略层(对比工具使用与直接推理)、轨迹层(滚动过程中破坏所有观测)和步骤层(在固定前缀下反事实替换单个观测)进行干预。步骤层估计量‘视觉证据增益’分离了每个观测的贡献。在六个代表性模型和五个细粒度感知基准上,发现策略校准错误,存在两种失效模式:‘未看即调用’——返回的观测对答案无因果影响;‘有看无规划’——观测信息有效但调用顺序混乱。轨迹层诊断表明,策略层准确率提升集中于少数校准良好的情况。这种现象被称为视觉工具使用的幻觉:尽管整体准确率有所提升,但在多数推理轨迹中,视觉工具并未产生实际因果效应。代码已公开于 https://github.com/OpenCausaLab/CauAudit。

原文摘要 · Abstract (English)

The "thinking-with-images" paradigm equips multimodal LLMs with active visual operations such as crop-and-zoom. However, models using these operations often achieve only marginal or negative gains over direct inference at substantially higher token cost. They may also repeatedly crop irrelevant regions and fail on questions that direct inference answers correctly. We ask whether the returned visual evidence causally affects the answer. To answer this question, we formulate visual tool-use as a causal graph that separates observation-mediated paths from action-induced shortcuts. We then audit it through interventions at the three levels: policy (comparing tool-use with direct inference), trajectory (corrupting all observations during rollout), and step (counterfactually replacing one individual observation under a fixed prefix). Our step-level estimand, Visual Evidence Gain, isolates the contribution of each returned observation. Across six representative models and five fine-grained perception benchmarks, we uncover policy miscalibration with two failure modes. In Calling Without Looking, returned observations have no causal effect on the answer. In Looking Without Planning, observations are informative but the call schedule is incoherent. A trajectory-level diagnostic decomposes the policy-level accuracy gain and shows that the gain is concentrated in a Calibrated minority. We term this discrepancy the illusion of visual tool-use: despite aggregate accuracy gains, visual tool-use is not causally effective across a broad range of rollouts. The code is available at https://github.com/OpenCausaLab/CauAudit.

多模态因果分析大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。