让AI推理时能精准定位图像区域,提升视觉理解能力。
Grounded Reinforcement Learning for Visual Reasoning
- 用强化学习让每步推理都对应图像具体坐标
- 在V*Bench上定位小元素准确率达86.4%
- 适合需要精细视觉定位的任务,如GUI操作
尽管基于思维链的强化学习已显著提升语言模型在数学和编程任务中的表现,但视觉推理需模型主动引导视觉注意、解析感知输入,并将抽象推理与空间证据结合。我们提出ViGoRL(视觉强化学习),一种通过强化学习训练的视觉-语言模型,显式将每一步推理锚定在特定视觉坐标上。受人类视觉决策启发,ViGoRL学习生成空间化推理轨迹,每步引导注意力至任务相关区域。当需要细粒度探索时,其新型多轮强化学习框架可动态聚焦预测坐标,实现逐步放大。在多个视觉推理基准上——包括SAT-2和BLINK(空间推理)、V*bench(视觉搜索)、ScreenSpot和VisualWebArena(网页定位)——ViGoRL均优于监督微调及缺乏显式接地机制的传统强化学习基线。引入多轮强化学习与缩放视觉反馈后,其在定位小型GUI元素和视觉搜索任务中表现显著提升,在V*Bench上达到86.4%准确率。此外,接地机制还增强了区域探索、接地子目标设定和视觉验证等行为。人工评估显示,模型的视觉参考不仅空间准确,且有助于理解推理过程。结果表明,视觉接地强化学习是赋予模型通用视觉推理能力的有效范式。
原文摘要 · Abstract (English)
While reinforcement learning (RL) over chains of thought has significantly advanced language models in tasks such as mathematics and coding, visual reasoning introduces added complexity by requiring models to direct visual attention, interpret perceptual inputs, and ground abstract reasoning in spatial evidence. We introduce ViGoRL (Visually Grounded Reinforcement Learning), a vision-language model trained with RL to explicitly anchor each reasoning step to specific visual coordinates. Inspired by human visual decision-making, ViGoRL learns to produce spatially grounded reasoning traces, guiding visual attention to task-relevant regions at each step. When fine-grained exploration is required, our novel multi-turn RL framework enables the model to dynamically zoom into predicted coordinates as reasoning unfolds. Across a diverse set of visual reasoning benchmarks--including SAT-2 and BLINK for spatial reasoning, V*bench for visual search, and ScreenSpot and VisualWebArena for web-based grounding--ViGoRL consistently outperforms both supervised fine-tuning and conventional RL baselines that lack explicit grounding mechanisms. Incorporating multi-turn RL with zoomed-in visual feedback significantly improves ViGoRL's performance on localizing small GUI elements and visual search, achieving 86.4% on V*Bench. Additionally, we find that grounding amplifies other visual behaviors such as region exploration, grounded subgoal setting, and visual verification. Finally, human evaluations show that the model's visual references are not only spatially accurate but also helpful for understanding model reasoning steps. Our results show that visually grounded RL is a strong paradigm for imbuing models with general-purpose visual reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。