通过深度排序任务分离视觉与语言对模型的影响,发现现有模型严重依赖语言线索。
Disentangling Pictorial Cue Understanding from Language Bias in VLMs via Depth Ordering Task

- 设计深度排序+奇数出列任务,控制视觉线索并测试语言清晰度影响
- 12个模型深度判断准确率仅47%-56%,均未超过随机水平
- 提出新度量方法揭示强语言偏见,适合关注模型可解释性的研究者
本文研究视觉语言模型(VLMs)的深度感知能力,旨在分离图像深度线索与语言干扰的影响。为此,结合深度排序与奇数出列心理物理学任务:模型需判断图像中一个物体相对于其他相同物体是更近还是更远。通过从模拟和真实3D场景生成2D视图,并控制单个图像深度线索的存在,实现对线索贡献的细粒度分析。语言影响通过指称表达清晰度变化来考察。我们引入一种新度量方法量化视觉与语言敏感性。基于该方法,构建包含3.7万张真实与合成图像及14.7万对图像-问题的奇数出列深度(O3-D)数据集。对12个开源与商用模型在O3-D上的评估显示,深度线索利用率低,深度排序准确率为47%至56%,无模型达到随机水平。同时,度量结果显示答案存在强烈语言偏见。链式思维(CoT)与上下文学习(ICL)均未显著提升性能,表明静态图像数据可能不足以支持深度理解。所有代码、图像生成流程及O3-D数据集已公开于https://github.com/lyiqian/o3-d。
原文摘要 · Abstract (English)
In this paper, we study depth perception of vision-language models (VLMs) to isolate the effects of pictorial depth cues and disentangle vision and language influences on model performance. To this end, we combine depth-ordering and odd-one-out psychophysical tasks: the VLMs are presented with images where one object is at different depth relative to other, otherwise identical, objects, and must determine whether the odd-one-out target is closer or farther to the observer. To create stimuli, we generate 2D views from simulated and real 3D scenes while controlling the presence of individual pictorial depth cues, enabling a fine-grained analysis of cue-level contributions. Language effects are examined by varying referring expression clarity. We also introduce a novel metric to quantify vision-vs-language sensitivities. Applying this methodology, we create the Odd-One-Out Depth (O3-D) dataset with 37K real and synthetic images and 147K image-question pairs. Evaluation of 12 open-source and commercial models on O3-D shows under-utilization of depth cues and depth-ordering accuracies between 47% and 56%, with no model above chance level. At the same time, our metric reveals strong linguistic bias in the answers. Neither chain-of-thought (CoT) nor in-context learning (ICL) significantly improves performance, suggesting that static image data alone may be insufficient for depth understanding. All code, the image generation pipeline, and the O3-D dataset are publicly released at https://github.com/lyiqian/o3-d.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。