发现视觉语言模型推理时对图像的访问有边界,长推理主要靠已有隐状态而非持续看图。
Visual Access Boundaries in Vision-Language Model Reasoning

- 通过遮蔽注意力路径,定位模型维持准确率所需的最小图像访问层
- 无论是否用思维链,模型在2层内就达到性能饱和,长推理不依赖持续看图
- 瓶颈是视觉信息读出难,而非计数能力不足,适合研究模型认知机制的人看
思维链(CoT)提示被广泛用于视觉语言模型(VLMs)的测试时扩展,但尚不清楚当生成更长推理轨迹时,模型究竟在扩展什么。我们探究了CoT是否需要持续访问图像标记,还是仅依赖前向传播中已提取的视觉信息。提出视觉访问扫除(Visual Access Sweep)方法,通过因果干预屏蔽生成标记查询对图像标记键的注意力,定义视觉访问边界(VAB)为维持任务准确率的最小访问区域。在六种Qwen2.5-VL和InternVL3模型配置中,无论是否使用无思维链直接回答,均存在有限的VAB。在Qwen2.5-VL-32B及InternVL3 14B与38B规模下,当对比完整访问目标时,使用思维链的模型其VAB层与无思维链边界相差不超过两层,尽管生成长度显著增加。这表明,思维链并非通过延长对图像标记的直接访问来提升性能,而是通过扩展对图像衍生隐状态的语言侧计算实现。进一步发现,思维链收益受感知读出能力限制:仅当所查视觉属性可被可靠读出时才有帮助,否则无效。符号属性模拟显示,一旦提供真实属性文本,思维链可改善计数;单对象探测-解码检查表明,某些强属性虽能从隐状态线性恢复,却难以由模型自身输出。综合分析表明,瓶颈在于读出而非计数。
原文摘要 · Abstract (English)
Chain-of-Thought (CoT) prompting is widely used as a test-time scaling strategy for Vision-Language Models (VLMs), but it remains unclear what is extended when VLMs generate longer reasoning traces. We ask whether CoT requires continued access to image tokens, or whether it mainly operates over visual information already made available earlier in the forward pass. We introduce Visual Access Sweep, a causal intervention that masks attention from generated-token queries to image-token keys along layer depth and generation time, and define the Visual Access Boundary (VAB) as the minimal access region that preserves task accuracy. Across six model configurations from Qwen2.5-VL and InternVL3, both no-CoT direct answering and CoT prompting exhibit finite VABs. In Qwen2.5-VL-32B and InternVL3 at 14B and 38B scales, when CoT is evaluated against the no-CoT full-access target, its VAB layer differs from the no-CoT boundary by at most two layers, despite substantially longer generations. This suggests that CoT does not primarily improve performance by prolonging direct image-token access throughout the reasoning trace, but by extending language-side computation over image-derived hidden-state information. We further show that CoT gains are constrained by perceptual readout. CoT helps when the queried visual attribute can be reliably read out by the model, but not when that readout is unreliable. A symbolic-attribute oracle shows that CoT can improve counting once ground-truth attributes are supplied as text, while a single-object probe-vs-decode check shows that hard attributes can be linearly recoverable from hidden states yet difficult for the model itself to output. Together, these analyses place the bottleneck at readout rather than counting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。