arXiv:2603.14117cs.CV2026-03被引 5

让视觉语言模型自动回看图像关键区域,提升推理准确性。

Improving Visual Reasoning with Iterative Evidence Refinement

  • 通过内部表示自动提取并重用图像显著区域特征
  • 在多个基准上平均提升8%的视觉推理性能
  • 无需外部图像操作,适合追求高精度推理的应用

视觉语言模型(VLMs)在图像推理方面能力日益增强,但稳健的视觉推理通常需要将中间步骤重新关联到原始视觉证据。现有方法多依赖外部图像操作(如缩放或裁剪)在推理中重新获取细粒度细节,这需要额外的图像重编码,可能打断推理流程。本文认为,VLMs 已具备识别和复用视觉证据的强内部信号,可直接利用。为此,提出端到端自回溯框架 SIEVE,训练模型通过内部表征重新激活图像证据。SIEVE 自动提取显著图像区域嵌入,并在需要时注入推理链,使后续步骤能基于相关视觉线索进行条件化,无需外部工具调用或重编码。采用强化学习指导模型何时触发视觉回溯及检索与插入哪些区域嵌入。在多个视觉推理基准上的实验,结合感知、推理与幻觉评估显示,SIEVE 实现一致提升,平均性能提高8%。

原文摘要 · Abstract (English)

Vision language models (VLMs) are increasingly capable of reasoning over images, but robust visual reasoning often requires re-grounding intermediate steps in the underlying visual evidence. Recent approaches typically rely on external image operations such as zooming or cropping to re-access fine-grained details during inference, which requires additional image re-encoding and can disrupt the reasoning trajectory. We argue that VLMs already provide strong internal signals for identifying and reusing visual evidence, and that these signals can be directly leveraged to support image-grounded reasoning. Motivated by this insight, we propose an end-to-end self-revisit framework, SIEVE, that trains models to re-engage image evidence through internal representations. SIEVE automatically extracts embeddings of salient image regions and injects them into the reasoning chain when additional grounding is needed, enabling later steps to condition on relevant visual cues without external tool calls or re-encoding. We use reinforcement learning to teach the model when to trigger visual revisiting and which region embeddings to retrieve and insert during the reasoning process. Experiments on multiple visual reasoning benchmarks, together with perception, reasoning, and hallucination evaluations, show that SIEVE yields consistent gains, improving performance by 8 percent on average across several benchmarks.

视觉推理自回溯强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。