通过引导视觉注意力与思维路径,让模型更聪明地看图推理。
Guiding the Inner Eye: A Framework for Hierarchical and Flexible Visual Grounded Reasoning
- 分两阶段训练,用认知启发信号引导模型聚焦关键物体和多样推理。
- 在TreeBench和V* Bench上达到开源模型最佳表现,显著提升复杂场景推理能力。
- 适合研究多模态推理、视觉理解与智能体决策的开发者和研究人员。
能够“以图思考”的模型通过动态将推理锚定在视觉证据上,代表了多模态AI的重大进展。然而,复现和推进这一能力颇具挑战,现有方法常陷入端到端强化学习(RL)的不稳定性与监督微调(SFT)的僵化性之间,导致模型要么难以学习,要么缺乏应对复杂真实场景所需的认知灵活性。为解决此困境,我们提出GRiP(Guided Reasoning and Perception)——一种新型两阶段训练框架,通过显式引导模型的感知焦点与逻辑路径,培养稳健且灵活的视觉接地推理能力。其核心在于认知增强型强化学习阶段,包含两项创新:(1) 重要性加权交并比奖励(Salience-Weighted IoU Reward),激励模型优先定位任务关键物体而非无关干扰物;(2) 多启发式奖励(Multi-Heuristic Reward),通过奖励多样且逻辑有效的推理路径,促进认知灵活性。基于Qwen2.5-VL-7B初始化,GRiP在多个挑战性基准测试中表现出显著性能提升,在高度复杂的TreeBench和V* Bench上取得开源模型最优结果,证明其在复杂视觉推理中的有效性。本工作表明,超越简单奖励机制,采用认知启发信号指导模型‘看什么’和‘如何想’,是释放下一代多模态智能的关键。代码将公开。
原文摘要 · Abstract (English)
Models capable of "thinking with images" by dynamically grounding their reasoning in visual evidence represent a major leap in multimodal AI. However, replicating and advancing this ability is non-trivial, with current methods often trapped between the instability of end-to-end reinforcement learning (RL) and the rigidity of supervised fine-tuning (SFT). This leads to models that either struggle to learn or lack the cognitive flexibility required for complex, real-world scenes. To navigate this dilemma, we introduce GRiP (Guided Reasoning and Perception), a novel two-stage training framework that cultivates robust and flexible visual grounded reasoning by explicitly guiding the model's perceptual focus and logical pathways. GRiP's core lies in its cognitive-enhanced RL stage, which features two key innovations: (1) a Salience-Weighted IoU Reward that incentivizes the model to prioritize the localization of mission-critical objects over trivial distractors, and (2) a Multi-Heuristic Reward that encourages cognitive flexibility by rewarding diverse yet logically valid reasoning pathways. Initialized from the Qwen2.5-VL-7B model, GRiP demonstrates significant performance gains across multiple challenging benchmarks. It achieves state-of-the-art results among open-source models on the highly challenging TreeBench and V* Bench, proving its effectiveness in complex visual reasoning. Our work demonstrates that moving beyond simplistic rewards and instead guiding models with cognitively-inspired signals for what to see and how to think is crucial for unlocking the next level of multimodal intelligence. The code will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。