arXiv:2608.30480cs.CVcs.LG2026-09

提出双视角检测方法,精准识别大模型中无图依据的物体幻觉。

VisER: Visual Evidence and Reliance for Object Hallucination Detection in LVLMs

论文配图:VisER: Visual Evidence and Reliance for Object Hallucination Detection in LVLMs
图 1 · 摘自论文原文
  • 从视觉证据和图像依赖双重角度评估生成物体的真实性
  • 在多个模型与基准上提升AUROC和AUPR指标
  • 无需额外生成,适合快速部署于现有视觉语言模型

物体幻觉仍是大型视觉语言模型中的可靠性难题,生成的物体描述可能看似合理却缺乏视觉依据。现有无训练检测器利用词元似然、注意力、视觉置信度或图文相似性等内部信号识别幻觉物体,但这些信号常受来源混淆影响:它们衡量的是模型内部对物体的支持强度,却无法区分支持来自特定物体的视觉证据,还是来自生成文本前缀。在复杂情况下,幻觉物体仍可能因符合场景、关联邻近视觉线索或自然衔接上下文而获得高内部支持。本文提出VisER,一种无训练的双视角物体级幻觉检测方法。该方法从两个互补维度评估每个生成物体:视觉证据衡量物体-上下文一致性是否由图像令牌中的特定物体证据支撑;视觉依赖衡量物体更多依赖图像而非生成前缀。二者结合生成更源感知的接地评分,且无需额外物体级验证生成。在多个LVLM和基准测试中,VisER显著优于多种基线方法,提升了AUROC和AUPR。

原文摘要 · Abstract (English)

Object hallucination remains a persistent reliability issue in large vision-language models, where generated object mentions may sound plausible but lack visual grounding. Recent training-free detectors use internal signals such as token likelihood, attention, visual confidence, or image-text similarity to identify hallucinated objects. These signals are useful, but they are often source-confounded. They measure how strongly an object is supported inside the model without distinguishing whether that support comes from object-specific visual evidence or the generated text prefix. In difficult cases, a hallucinated object can still receive high internal support because it fits the scene, is associated with nearby visual cues, or follows naturally from the generated text prefix. We propose VisER, a training-free two-sided metric for object-level hallucination detection. VisER evaluates each generated object mention from two complementary views. Visual Evidence measures whether object-context compatibility is backed by object-specific evidence from image tokens. Visual Reliance measures whether the object is supported more by the image than by the generated prefix. Combining these views gives a more source-aware grounding score, while avoiding additional object-level verification generations. Across multiple LVLMs and benchmarks, VisER improves AUROC and AUPR over a range of baselines.

幻觉检测视觉语言模型无训练图像依赖

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。