揭示视觉语言模型属性幻觉的根源并提出精准修复方法
When Visual Signals Mislead: A Mechanistic Study of Attribute Hallucination in Vision-Language Models
- 通过分离视觉与语言信号，定位属性错误的真正原因
- 实验证明视觉信号才是错误主因，语言先验并非主因
- 针对不同错误类型自动匹配修复策略，无需依赖先验假设
属性幻觉——即视觉语言模型（VLMs）正确识别物体但错误描述其属性——普遍存在，但机制尚不明确。主流解释‘语言先验主导’未在属性层面直接验证。本文提出VISOR（视觉操作修复）框架，结合零图像诊断与路由修复。其VSNR诊断将每个预测分解为视觉logit信号和语言先验信号。在来自三个VLM家族、三种属性类型的10,791个负样本上，视觉信号显著预测假阳性，而语言先验信号接近随机水平。VISOR据此区分两类失败模式：颜色/状态属性中低置信但方向正确的视觉信号，以及材质属性中低信噪比或错位的视觉信号。相应地，系统自动调用校准、放弃决策或定向视觉适配等操作器进行修复。在Qwen、InternVL和LLaVA上，VISOR有效降低属性假阳性，且不依赖语言先验主导假设。
原文摘要 · Abstract (English)
Attribute hallucination---where vision-language models (VLMs) correctly identify an object but mischaracterize its properties---is prevalent yet mechanistically poorly understood. The dominant explanation, language-prior dominance, has motivated prior-suppression methods, but this explanation has not been directly tested at the attribute level. We present VISOR (Visual-Operational Remediation), a unified framework that couples null-image-based diagnosis with routed remediation. Its VSNR diagnostic decomposes each prediction into a visual logit signal and a language-prior signal. Across 10,791 negative-ground-truth samples from three VLM families and three attribute types, the visual signal strongly predicts false positives, whereas the language-prior signal is near chance. VISOR uses this diagnosis to separate two failure modes: low-margin but directionally correct visual signals in color/state attributes, and low-SNR or misaligned visual signals in material attributes. The same diagnosis routes each query to the appropriate operator: calibration for threshold-placement errors, abstention for training-free low-SNR handling, or targeted visual adaptation for material failures that prior suppression cannot correct. Across Qwen, InternVL, and LLaVA, VISOR reduces attribute false positives without relying on the prior-dominance assumption.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。