测试视觉语言模型的视角理解能力，发现其表现显著下降且与检测任务无关。
Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Models
- 构建两个手工标注数据集，专门评估模型视角理解能力。
- 12个主流模型在需要视角转换时性能大幅下降。
- 现有检测任务无法有效衡量模型的视角理解水平。
视觉视角理解（VPT）指理解他人视角的能力，使个体能够预测他人行为。例如，司机可通过判断行人所见来避免事故。人类通常在童年早期就发展出该能力，但近期兴起的视觉语言模型（VLMs）是否具备此能力尚不明确。随着这些模型在真实世界中广泛应用，了解其在复杂任务如VPT上的表现变得至关重要。本文提出两个手动构建的数据集：Isle-Bricks和Isle-Dots，用于评估12种常用VLMs的VPT能力。结果显示，所有模型在需要视角转换时性能显著下降。此外，物体检测任务的表现与VPT任务表现相关性很低，表明现有基准可能不足以全面评估该能力。代码与数据集将公开于 https://sites.google.com/view/perspective-taking。
原文摘要 · Abstract (English)
Visual perspective-taking (VPT), the ability to understand the viewpoint of another person, enables individuals to anticipate the actions of other people. For instance, a driver can avoid accidents by assessing what pedestrians see. Humans typically develop this skill in early childhood, but it remains unclear whether the recently emerging Vision Language Models (VLMs) possess such capability. Furthermore, as these models are increasingly deployed in the real world, understanding how they perform nuanced tasks like VPT becomes essential. In this paper, we introduce two manually curated datasets, Isle-Bricks and Isle-Dots for testing VPT skills, and we use it to evaluate 12 commonly used VLMs. Across all models, we observe a significant performance drop when perspective-taking is required. Additionally, we find performance in object detection tasks is poorly correlated with performance on VPT tasks, suggesting that the existing benchmarks might not be sufficient to understand this problem. The code and the dataset will be available at https://sites.google.com/view/perspective-taking
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。