arXiv:2608.03817cs.CVcs.AI2026-08

发现大模型幻觉有独特模式,用双重维度检测更准。

UHP Detection: LVLMs have their Unique Hallucination Pattern in the Consistency Space

论文配图:UHP Detection: LVLMs have their Unique Hallucination Pattern in the Consistency Space
图 1 · 摘自论文原文
  • 从图像与文本扰动、正反逻辑两维建模幻觉模式。
  • 在多个数据集上比现有方法最高提升18.72%的检出率。
  • 适合研究模型可靠性或开发抗幻觉系统的开发者。

大型视觉-语言模型(LVLMs)虽具备强大多模态推理能力,但仍易产生不基于视觉证据的幻觉。现有黑盒幻觉检测方法依赖单一一致性度量,隐含假设模型不确定性可由单一指标表征,但幻觉在不同行为探针中呈现多样化的不确定性表现,单一指标难以充分刻画其底层行为。本文提出全黑盒框架「独特幻觉模式(UHP)检测」,将幻觉建模为两个维度定义的结构化不确定性模式:扰动模态(图像 vs. 文本)与逻辑极性(陈述 vs. 其否定)。二者交集生成四个互补的一致性组,从中提取组内与组间特征,训练轻量分类器。在AMBER和PhD数据集上对三种LVLM进行综合实验,UHP检测始终优于先前黑盒与白盒基线,相较最强黑盒方法,AUC-ROC最高提升18.72%,AUC-PR最高提升20.07%。大量消融实验表明各一致性组提供互补信息,组合形成结构化幻觉模式。跨数据集评估显示该模式具备良好泛化能力,表明幻觉行为反映模型特有的一致性模式。代码已公开于 https://github.com/amirezzati/uhpdet。

原文摘要 · Abstract (English)

Large vision--language models (LVLMs) demonstrate strong multimodal reasoning capabilities but remain prone to hallucination, where model predictions are not grounded in visual evidence. Existing black-box hallucination detection methods estimate uncertainty through a single consistency metric, implicitly assuming that model uncertainty can be adequately characterized by a single measure. However, hallucinations exhibit diverse manifestations of uncertainty across different behavioral probes, making a single measure insufficient to characterize their underlying behavior. We propose \emph{Unique Hallucination Pattern (UHP) Detection}, a fully black-box framework that models hallucination as a structured uncertainty pattern defined by two axes: perturbation modality (image vs.\ text) and logical polarity (a statement vs.\ its negation). Their intersection produces four complementary consistency groups that capture distinct manifestations of model uncertainty, from which both within-group and between-group features are extracted to train a lightweight classifier. Through comprehensive experiments on AMBER and PhD across three LVLMs, UHP Detection consistently outperforms prior black-box and white-box baselines, with improvements of up to $+18.72\%$ AUC-ROC and $+20.07\%$ AUC-PR over the strongest black-box methods. Extensive ablation studies demonstrate that each consistency group contributes complementary information and that their combination forms a structured hallucination pattern. Furthermore, cross-dataset evaluation shows that this learned pattern generalizes across benchmarks, indicating that hallucination behavior reflects a model-specific consistency pattern. \textbf{Code is publicly available at} https://github.com/amirezzati/uhpdet.

幻觉检测多模态一致性分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。