用可控模糊图像测试人与机器对视觉概念的判断边界。
A Framework for Generating Semantically Ambiguous Images to Probe Human and Machine Perception
- 在CLIP嵌入空间中插值生成语义模糊图像
- 机器更倾向识别为兔子,人类更接近合成基准
- 引导系数对人类感知影响大于模型
经典鸭兔错觉表明,当视觉证据模糊时,人脑必须决定看到什么。但人类观察者在‘鸭子’与‘兔子’之间画出的界限究竟在哪里?机器分类器是否在相同位置做出判断?我们使用语义模糊图像作为可解释性探针,揭示视觉模型如何表征概念之间的边界。提出一种受心理物理学启发的框架,在CLIP嵌入空间中插值生成连续的模糊图像谱,精确测量人类与机器分类器的语义边界位置。实验发现,机器分类器更倾向于将图像识别为‘兔子’,而人类则更贴近用于合成的CLIP嵌入基准;此外,引导系数对人类感知的影响显著强于对机器分类器的影响。该框架展示了可控模糊如何作为诊断工具,弥合人类心理物理分析、图像分类与生成模型之间的差距,为人类-模型对齐、鲁棒性、模型可解释性及图像合成方法提供洞见。
原文摘要 · Abstract (English)
The classic duck-rabbit illusion reveals that when visual evidence is ambiguous, the human brain must decide what it sees. But where exactly do human observers draw the line between ''duck'' and ''rabbit'', and do machine classifiers draw it in the same place? We use semantically ambiguous images as interpretability probes to expose how vision models represent the boundaries between concepts. We present a psychophysically-informed framework that interpolates between concepts in the CLIP embedding space to generate continuous spectra of ambiguous images, allowing us to precisely measure where and how humans and machine classifiers place their semantic boundaries. Using this framework, we show that machine classifiers are more biased towards seeing ''rabbit'', whereas humans are more aligned with the CLIP embedding used for synthesis, and the guidance scale seems to affect human sensitivity more strongly than machine classifiers. Our framework demonstrates how controlled ambiguity can serve as a diagnostic tool to bridge the gap between human psychophysical analysis, image classification, and generative image models, offering insight into human-model alignment, robustness, model interpretability, and image synthesis methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。