通过融合多个内部信号，提升视觉语言模型幻觉检测的准确性与鲁棒性。
EnsemHalDet: Robust VLM Hallucination Detection via Ensemble of Internal State Detectors
- 利用注意力输出和隐藏状态等多重内部表征构建检测器
- 在多个VQA数据集上AUC表现优于现有方法
- 适合需要高可靠性的多模态应用开发者
视觉语言模型（VLMs）在多模态任务中表现优异，但仍易产生事实错误或脱离输入图像的幻觉。近期研究发现，基于内部表征的幻觉检测比仅依赖模型输出的方法更高效、准确。然而，现有方法通常仅使用单一表征或检测器，难以捕捉多样化的幻觉信号。本文提出EnsemHalDet，一种基于集成学习的幻觉检测框架，利用VLMs的多种内部表征（包括注意力输出和隐藏状态），为每种表征训练独立检测器，并通过集成方式融合结果。在多个VQA数据集和VLM模型上的实验表明，EnsemHalDet在AUC指标上持续优于先前方法及单检测器模型，证明集成多样化内部信号能显著提升多模态幻觉检测的鲁棒性。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) excel at multimodal tasks, but they remain vulnerable to hallucinations that are factually incorrect or ungrounded in the input image. Recent work suggests that hallucination detection using internal representations is more efficient and accurate than approaches that rely solely on model outputs. However, existing internal-representation-based methods typically rely on a single representation or detector, limiting their ability to capture diverse hallucination signals. In this paper, we propose EnsemHalDet, an ensemble-based hallucination detection framework that leverages multiple internal representations of VLMs, including attention outputs and hidden states. EnsemHalDet trains independent detectors for each representation and combines them through ensemble learning. Experimental results across multiple VQA datasets and VLMs show that EnsemHalDet consistently outperforms prior methods and single-detector models in terms of AUC. These results demonstrate that ensembling diverse internal signals significantly improves robustness in multimodal hallucination detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。