测试视觉语言模型能否发现人一眼就能看出的视觉异常
Vision-Language Models Can't See the Obvious
- 构建新基准，检测模型对颜色、亮度、方向等基础视觉特征的感知能力
- 顶级模型如GPT-4o在简单任务中准确率仅47.6%
- 适合关注模型视觉理解缺陷的研究者和开发者
我们提出视觉显著性基准（SalBench），用于评估大视觉语言模型（LVLM）对人类显而易见的视觉显著特征的识别能力，例如在一组小圆中突出的大圆。该基准聚焦于颜色、强度和方向等低层视觉特征，这些特征是人类视觉处理的基础。SalBench包含三类新任务：奇数项识别、指代奇数项识别与视觉指代奇数项识别，图像设计突出场景中罕见、异常或意外元素，自然吸引人类注意力。我们对当前最先进的LVLM进行全面评估，发现令人惊讶的局限性：即使先进模型如GPT-4o在如此简单的任务上准确率也仅为47.6%。SalBench将为衡量与人类注意机制契合的LVLM能力提供重要参考。
原文摘要 · Abstract (English)
We present Saliency Benchmark (SalBench), a novel benchmark designed to assess the capability of Large Vision-Language Models (LVLM) in detecting visually salient features that are readily apparent to humans, such as a large circle amidst a grid of smaller ones. This benchmark focuses on low-level features including color, intensity, and orientation, which are fundamental to human visual processing. Our SalBench consists of images that highlight rare, unusual, or unexpected elements within scenes, and naturally draw human attention. It comprises three novel tasks for evaluating the perceptual capabilities of LVLM: Odd-One-Out Detection, Referring Odd-One-Out, and Visual Referring Odd-One-Out. We perform a comprehensive evaluation of state-of-the-art LVLM using SalBench and our findings reveal a surprising limitation: LVLM struggle to identify seemingly obvious visual anomalies, with even the advanced GPT-4o achieving only 47.6\% accuracy on such a simple task. SalBench will be an important step in measuring the capabilities of LVLM that align with the subtle definition of human attention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。