让AI推理时标注图像区域,提升思维可验证性。
Thinking with Visual Grounding

- 推理过程穿插语言与图像区域标注,显式关联视觉证据。
- 在计数和空间推理任务上,4B模型性能接近甚至超越27B模型。
- 点标注适合计数,框标注结合奖励机制更优空间推理。
视觉推理不应仅听起来合理,还应提供图像证据。现有视觉语言模型虽能生成自然语言推理链,但常隐含支持图像区域,难以验证与监督。本文提出视觉接地推理:模型在语言思考中穿插显式的点或框标注,明确对应图像中的关键对象。为此,我们构建可扩展合成流程,通过蒸馏正确推理链、提取所需视觉对象、用SAM3代理进行标注,并生成对齐的点/框监督信号。进一步提出接地感知强化学习,结合答案正确性奖励与密集接地奖励(评估物体引用是否匹配真实图像证据)。在两个计数基准和四个空间推理基准上,将视觉接地推理加入Gemma3-4B-IT后,性能持续优于原模型与无接地基线。在空间推理任务中,4B模型表现媲美甚至超过同系列的27B模型。分析表明,点标注适用于计数任务,而框标注在获得显式接地奖励时显著提升空间推理能力。结果证明,当中间思考与图像区域绑定时,模型推理更准确。
原文摘要 · Abstract (English)
Visual thinking should not only sound right; it should show its evidence. While recent vision-language models (VLMs) can produce natural-language reasoning traces, these traces often leave the supporting image regions implicit, making them hard to verify and difficult to supervise. We introduce visually grounded thinking, a reasoning process in which models interleave natural-language thoughts with explicit point or box groundings of the visual evidence used at each step. This lets the model express intermediate reasoning in language while grounding key objects in the image regions they refer to. To train this behavior, we construct a scalable synthesis pipeline that distills correct visual reasoning traces, extracts the visual objects required by the traces, grounds them with a SAM3-based agent, and derives aligned point and box supervision from the resulting masks. We further propose grounding-aware reinforcement learning, which combines answer correctness rewards with dense grounding rewards that score whether generated object references match the correct image evidence. Across two counting benchmarks and four spatial reasoning benchmarks, adding visually grounded thinking to Gemma3-4B-IT consistently improves performance over the original model and the non-grounded thinking baseline. On spatial reasoning, the visually grounded thinking 4B models match, and in some cases surpass, Gemma3-27B-IT from the same model family. Our analysis shows that point grounding is well suited to counting, while box grounding benefits most from explicit grounding rewards on spatial tasks. Overall, our results show that VLMs think better when their intermediate thoughts are tied to the image regions that make them true.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。