用错视人脸测试模型在模糊图像下的判断能力,发现大模型易误判情绪。
When Visual Evidence is Ambiguous: Pareidolia as a Diagnostic Probe for Vision Models
- 用错视人脸作为诊断工具,分析模型对模糊视觉的响应机制
- LLaVA在73%非人脸图像中错误识别人脸,尤其对负面情绪过度反应
- 不同模型策略差异大:大模型过激,纯视觉模型保守,检测模型最稳
当视觉证据模糊时,视觉模型需决定如何解释类人脸模式。错视人脸(在非人脸物体中感知人脸)为这类决策提供了受控诊断工具。我们提出一个分析框架,评估六种模型在类别、难度和情绪维度上的检测、定位、不确定性与偏差表现。模型涵盖四类表征范式:视觉语言模型(CLIP-B/32, CLIP-L/14, LLaVA-1.5-7B)、纯视觉分类(ViT)、目标检测(YOLOv8)和人脸检测(RetinaFace)。结果表明,不确定性和偏差可解耦:低不确定性可能代表安全抑制(如检测器),也可能体现极端过度解读(如视觉语言模型)。视觉语言模型存在语义过激活,系统性地将模糊非人类区域视为人类,其中LLaVA在73%的非人类错视图像上误报,尤其在负向情绪场景下更显著。ViT则采用不确定性即回避策略,保持模糊但基本无偏。基于检测的模型通过保守先验抑制错视反应,即使定位控制也有效。这些结果说明,模糊下的行为更多由表征方式决定而非阈值,确立错视人脸作为语义鲁棒性的诊断工具,并提供面向模糊的难例样本用于模型训练。代码将在发表后公开。
原文摘要 · Abstract (English)
When visual evidence is ambiguous, vision models must decide how to interpret face-like patterns. Face pareidolia, the perception of faces in non-face objects, provides a controlled probe of such decisions. We introduce a diagnostic framework that analyzes detection, localization, uncertainty and bias across class, difficulty and emotion. We evaluate six models spanning four representational regimes: vision-language models (VLMs; CLIP-B/32, CLIP-L/14, LLaVA-1.5-7B), pure vision classification (ViT), object detection (YOLOv8), and face detection (RetinaFace). Our results reveal that uncertainty and bias are decoupled: low uncertainty can signal either safe suppression, as in detectors, or extreme over-interpretation, as in VLMs. VLMs exhibit semantic overactivation, systematically interpreting ambiguous non-human regions as Human, with LLaVA over-calling on 73% of non-human pareidolic images, especially for negative emotions. ViT instead follows an uncertainty-as-abstention strategy, remaining diffuse yet largely unbiased. Detection-based models achieve low bias through conservative priors that suppress pareidolia responses even when localization is controlled. Together, these results show that behavior under ambiguity is governed more by representation than thresholds, establishing face pareidolia as a diagnostic of semantic robustness and a source of ambiguity-aware hard negatives for vision models. Code will be released upon publication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。