让大模型在不依赖显式视觉定位的情况下,也能精准推理图像细节。
iVGR: Internalizing Visually Grounded Reasoning for MLLMs with Reinforcement Learning

- 用强化学习把视觉定位能力内化到文本推理中,不再需要显式框选物体。
- 在细粒度图像理解任务上超越现有方法,准确率显著提升。
- 适合需要高效推理且支持工具调用的多模态应用开发者。
尽管基于视觉的思维链(CoT)已成为提升多模态大模型细粒度感知能力的有前景范式,但其在推理阶段的效果仍缺乏深入探索。本文实证发现,在推理时强制要求显式物体框会降低性能,相比不依赖视觉定位的纯文本思维链。我们提出内部化视觉推理(iVGR),一种新型强化学习框架,将定位能力融入文本推理过程。采用双流训练策略,通过一致性奖励使文本流与高质量视觉引导流对齐,实现推理时无需显式视觉定位即可精准定位。大量实验表明,该方法在细粒度基准测试上显著优于现有基线,同时保持支持工具辅助推理流程的灵活性。
原文摘要 · Abstract (English)
While visually grounded Chain-of-Thought (CoT) has emerged as a promising paradigm to enhance fine-grained perception in multimodal large language models (MLLMs), its efficacy during the inference phase remains underexplored. In this work, we empirically find that mandating explicit object boxes in visually grounded CoT during inference often degrades performance compared to standard textual CoT, which reasons without explicit visual grounding. We hypothesize that the visual localization capability can be internalized into the textual CoT and that the mandatory explicit grounding introduces unnecessary interference with the model's primary objective of answer prediction. To address this problem, we propose Internalizing Visually Grounded Reasoning (\textbf{iVGR}), a novel reinforcement learning framework that transfers localization capabilities into the textual reasoning process. We employ a dual-stream training strategy, where a textual stream is aligned with a high-quality visually grounded stream via a proposed consistency reward, enabling the model to localize accurately without explicit grounding during inference. Extensive experiments demonstrate that our method significantly outperforms existing baselines on fine-grained benchmarks, while maintaining the flexibility to support tool-assisted inference workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。