让AI像人一样主动聚焦,用内建目光控制提升视觉推理能力
GazeVLM: Active Vision via Internal Attention Control for Multimodal Reasoning

- 模型自动生成注视标记,动态控制注意力焦点
- 在高分辨率任务上超越同类模型近4%,比图像思维流水线高5%以上
- 无需外部工具或扩大上下文,实现全局与局部推理无缝切换
人类视觉推理依赖主动视觉,即元认知控制驱动的自上而下注意力,动态将中心视区聚焦于任务相关细节,同时保持对全局场景的周边感知。相比之下,现代视觉语言模型(VLMs)被动处理视觉信息,依赖大量静态令牌上下文积累,削弱空间推理并引发语言幻觉。本文提出范式转变:GazeVLM,一种将元认知注意力控制内置于推理循环的多模态架构。通过自主生成注视令牌(<LOOK>),GazeVLM建立对自身因果注意力掩码的自上而下控制机制,动态决定关注意图,触发对无关视觉特征的持续抑制,实现空间选择性注意并模拟中央凹聚焦。局部推理完成后,抑制解除,无缝恢复全局视野。该架构使模型在不依赖外部代理工具(如裁剪工具)或扩充上下文窗口的情况下,实现全局意识与局部聚焦推理的流畅切换。采用定制的组相对策略优化(GRPO)训练,40亿参数的GazeVLM在HRBench-4k和HRBench-8k上表现优于同参数量级的最先进VLMs近4%,且在基于图像思考的代理流水线上提升超5%。
原文摘要 · Abstract (English)
Human visual reasoning is governed by active vision, a process where metacognitive control drives top-down goal-directed attention, dynamically routing foveal focus toward task-relevant details while maintaining peripheral awareness of the global scene. In contrast, modern Vision-Language Models (VLMs) process visual information passively, relying on the static accumulation of massive token contexts that dilute spatial reasoning and induce linguistic hallucinations. Here we propose the following paradigm shift: GazeVLM, a multimodal architecture that internalizes this metacognitive oversight over its deployment of attention resources directly into the reasoning loop. By empowering the VLM to autonomously generate gaze tokens ($\texttt{<LOOK>}$), GazeVLM establishes a top-down control mechanism over its own causal attention mask. The model dynamically dictates its focal intent, triggering a continuous suppression bias that dampens irrelevant visual features, implementing spatial selective attention and simulating foveal fixation. Once local reasoning concludes, the bias lifts, seamlessly restoring the global view. This architecture enables the model to fluidly transition between global spatial awareness and localized focal reasoning without relying on external agentic contraptions like cropping tools, or inflating the context window with additional visual tokens derived from localized visual patches. Trained with a bespoke Group Relative Policy Optimization (GRPO) procedure that rewards valid grounding, our 4B-parameter GazeVLM delivers strong high-resolution multimodal reasoning performance, surpassing state-of-the-art VLMs in its parameter class by nearly 4% and agentic multimodal pipelines built around thinking with images by more than 5% on HRBench-4k and HRBench-8k.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。