测试视觉语言模型对被遮挡物体的空间推理能力，发现当前模型表现不佳。
CAPTURe: Evaluating Spatial Reasoning in Vision Language Models via Occluded Object Counting
- 设计新任务CAPTURe，通过遮挡图案推断隐藏物体数量
- 最强模型GPT-4o在遮挡情况下计数错误率显著上升
- 人类几乎无误，提示模型缺乏空间补全能力
识别和推理被遮挡（部分或完全隐藏）物体对于理解视觉场景至关重要，因为遮挡在真实环境中频繁出现，阻碍空间理解。为评估模型对多个遮挡物体的推理能力，我们提出新任务CAPTURe（Counting Amodally for Patterns Through Unseen REgions），要求模型通过推断模式如何在遮挡物后延续来计数排列成规律的物体。CAPTURe需同时识别视觉模式并进行推理，是检验视觉语言模型（VLMs）是否理解遮挡模式及具备空间理解能力的有效基准。该任务还测试模型构建世界模型以填补缺失信息的能力。CAPTURe包含两部分：(1) CAPTURe-real，由人工筛选的真实物体图像组成；(2) CAPTURe-synthetic，生成的受控诊断数据集。我们在四个强VLM（GPT-4o、Intern-VL2、Molmo、Qwen2-VL）上评估，发现模型在遮挡与未遮挡模式下均表现不佳。关键发现：模型在遮挡情况下的表现更差，表明其在推断未见空间关系方面存在缺陷，即使是最强的GPT-4o也无法准确计数。相比之下，人类在该任务中误差极小。此外，提供遮挡物位置的辅助信息可提升模型性能，说明错误源于对遮挡处理和图像计数双重困难。代码与数据：https://github.com/atinpothiraj/CAPTURe
原文摘要 · Abstract (English)
Recognizing and reasoning about occluded (partially or fully hidden) objects is vital to understanding visual scenes, as occlusions frequently occur in real-world environments and act as obstacles for spatial comprehension. To test models' ability to reason about multiple occluded objects, we introduce a novel task, Counting Amodally for Patterns Through Unseen REgions (CAPTURe), which requires a model to count objects arranged in a pattern by inferring how the pattern continues behind an occluder (an object which blocks parts of the scene). CAPTURe requires both recognizing visual patterns and reasoning, making it a useful testbed for evaluating vision-language models (VLMs) on whether they understand occluded patterns and possess spatial understanding skills. By requiring models to reason about occluded objects, CAPTURe also tests VLMs' ability to form world models that would allow them to fill in missing information. CAPTURe consists of two parts: (1) CAPTURe-real, with manually filtered images of real objects in patterns and (2) CAPTURe-synthetic, a controlled diagnostic with generated patterned images. We evaluate four strong VLMs (GPT-4o, Intern-VL2, Molmo, and Qwen2-VL) on CAPTURe, finding that models struggle to count on both occluded and unoccluded patterns. Crucially, we find that models perform worse with occlusion, suggesting that VLMs are also deficient in inferring unseen spatial relationships: even the strongest VLMs like GPT-4o fail to count with occlusion. In contrast, we find that humans achieve very little error on CAPTURe. We also find that providing auxiliary information of occluded object locations increases performance, underscoring that the model error comes both from an inability to handle occlusion as well as difficulty in counting in images. Code and data: https://github.com/atinpothiraj/CAPTURe
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。