arXiv:2506.22146cs.CVcs.AI2025-06NeurIPS被引 11

通过添加空间结构增强视觉输入,显著提升大模型的视觉推理能力

Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs

  • 在视觉输入中加入低级空间结构,引导模型进行序列化空间感知
  • 在视觉搜索、计数和空间关系任务上分别提升25.0%、26.8%、9.5%
  • 强调视觉结构设计比纯文本提示更关键,适合需要空间推理的研究者

尽管大型视觉语言模型(LVLMs)取得进展,其视觉推理能力仍受限于绑定问题:难以可靠地将感知特征与正确的视觉指代关联。这一缺陷导致计数、视觉搜索、场景描述和空间关系理解等任务中持续出错。核心原因在于当前模型大多并行处理视觉特征,缺乏空间定位的串行注意力机制。本文提出视觉输入结构增强推理(VISER),通过在视觉输入中加入低级空间结构,并搭配引导序列化、空间感知解析的文本提示,仅用单次查询推理即实现显著性能提升。实验表明,VISER使GPT-4o在视觉搜索、计数和空间关系任务上的表现分别提升25.0%、26.8%和9.5%,在2D数据集上将场景描述的编辑距离误差降低0.32。进一步发现,仅靠文本策略(如思维链提示)无法带来增益,甚至会降低性能。结果表明,视觉输入设计比纯语言推理策略更为重要,视觉结构化是提升组合与空间推理能力的有效通用方法。

原文摘要 · Abstract (English)

Despite progress in Large Vision-Language Models (LVLMs), their capacity for visual reasoning is often limited by the binding problem: the failure to reliably associate perceptual features with their correct visual referents. This limitation underlies persistent errors in tasks such as counting, visual search, scene description, and spatial relationship understanding. A key factor is that current LVLMs process visual features largely in parallel, lacking mechanisms for spatially grounded, serial attention. This paper introduces Visual Input Structure for Enhanced Reasoning (VISER), a simple, effective method that augments visual inputs with low-level spatial structures and pairs them with a textual prompt that encourages sequential, spatially-aware parsing. We empirically demonstrate substantial performance improvements across core visual reasoning tasks, using only a single-query inference. Specifically, VISER improves GPT-4o performance on visual search, counting, and spatial relationship tasks by 25.0%, 26.8%, and 9.5%, respectively, and reduces edit distance error in scene description by 0.32 on 2D datasets. Furthermore, we find that the visual modification is essential for these gains; purely textual strategies, including Chain-of-Thought prompting, are insufficient and can even degrade performance. VISER underscores the importance of visual input design over purely linguistically based reasoning strategies and suggests that visual structuring is a powerful and general approach for enhancing compositional and spatial reasoning in LVLMs.

视觉推理空间结构多模态模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。