提出几何熵方法，精准检测视觉语言模型的幻觉问题。
HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision-Language Models
- 通过扰动图像与答案聚类，构建统一幻觉检测框架。
- 在多个数据集上验证，密集视觉编码模型幻觉更易识别。
- 适合关注多模态模型可靠性与输出可信度的研究者。
视觉语言模型（VLMs）虽能实现开放域视觉问答，但仍易产生幻觉。本文提出HEDGE，一种整合可控视觉扰动、语义聚类与鲁棒不确定性度量的统一幻觉检测框架。该框架将采样、畸变生成、聚类（蕴含与嵌入基础）及度量计算集成到可复现的流程中，适用于多种多模态架构。在VQA-RAD和KvasirVQA-x1数据集上，对LLaVA-Med、Med-Gemma、Qwen2.5-VL三类代表性模型的评估显示，幻觉可检测性与模型架构和提示设计密切相关：采用密集视觉标记的统一融合模型（如Qwen2.5-VL）表现最佳，受限标记架构（如Med-Gemma）最差。直接对生成答案应用嵌入聚类通常分离效果更优，而蕴含聚类对LLaVA-Med及长句回答仍具优势。总体而言，VASE度量在各种配置下均提供最稳定的幻觉信号，尤其在嵌入聚类与中等采样预算（n ~ 10-15）下表现突出。提示设计亦影响显著：简洁的标签式输出比语法受限的单句回复更具清晰语义结构。通过将幻觉检测建模为由采样规模、提示结构、模型架构与聚类策略共同决定的几何鲁棒性问题，HEDGE为多模态可靠性评估提供了原理性强、计算感知明确的基础。相关工具库hedge-bench已发布于PyPI，完整代码与实验资源见https://github.com/Simula/HEDGE。
原文摘要 · Abstract (English)
Vision-language models (VLMs) enable open-ended visual question answering but remain prone to hallucinations. We present HEDGE, a unified framework for hallucination detection that combines controlled visual perturbations, semantic clustering, and robust uncertainty metrics. HEDGE integrates sampling, distortion synthesis, clustering (entailment- and embedding-based), and metric computation into a reproducible pipeline applicable across multimodal architectures. Evaluations on VQA-RAD and KvasirVQA-x1 with three representative VLMs (LLaVA-Med, Med-Gemma, Qwen2.5-VL) reveal clear architecture- and prompt-dependent trends. Hallucination detectability is highest for unified-fusion models with dense visual tokenization (Qwen2.5-VL) and lowest for architectures with restricted tokenization (Med-Gemma). Embedding-based clustering often yields stronger separation when applied directly to the generated answers, whereas NLI-based clustering remains advantageous for LLaVA-Med and for longer, sentence-level responses. Across configurations, the VASE metric consistently provides the most robust hallucination signal, especially when paired with embedding clustering and a moderate sampling budget (n ~ 10-15). Prompt design also matters: concise, label-style outputs offer clearer semantic structure than syntactically constrained one-sentence responses. By framing hallucination detection as a geometric robustness problem shaped jointly by sampling scale, prompt structure, model architecture, and clustering strategy, HEDGE provides a principled, compute-aware foundation for evaluating multimodal reliability. The hedge-bench PyPI library enables reproducible and extensible benchmarking, with full code and experimental resources available at https://github.com/Simula/HEDGE .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。