通过语义聚类与时空扰动,用熵值检测视频模型幻觉。
VideoHEDGE: Entropy-Based Hallucination Detection for Video-VLMs via Semantic Clustering and Spatiotemporal Perturbations
- 基于语义聚类和多扰动生成,计算文本输出的熵值可靠性分数。
- 在足球视频问答上,VASE得分在大扰动下仍保持最高检出率。
- 方法轻量高效,适合评估视频多模态模型的可信度。
视频视觉语言模型(Video-VLMs)中的幻觉问题频发且置信度高,现有不确定性度量常与正确性不一致。本文提出 VideoHEDGE,一个模块化框架,将图像级熵值可靠性估计拓展至时序结构输入。给定视频-问题对,VideoHEDGE 从原始片段及光度与时空扰动后的变体中生成多个高温答案,使用自然语言推理(NLI)或嵌入聚类法将输出文本聚类为语义假设,进而计算三类可靠性分数:语义熵(SE)、RadFlag 和视觉增强语义熵(VASE)。在 SoccerChat 基准上,采用大语言模型作为评判者获得二元幻觉标签。对三个 7B 视频模型(Qwen2-VL、Qwen2.5-VL 及足球微调模型)测试显示,VASE 在较大扰动预算下始终表现最优,而 SE 与 RadFlag 常接近随机水平。嵌入聚类法性能媲美 NLI 聚类,但计算成本显著更低;领域微调虽降低幻觉频率,但校准改进有限。相关工具包 hedge-bench 已开源,代码与实验资源见 https://github.com/Simula/HEDGE#videohedge。
原文摘要 · Abstract (English)
Hallucinations in video-capable vision-language models (Video-VLMs) remain frequent and high-confidence, while existing uncertainty metrics often fail to align with correctness. We introduce VideoHEDGE, a modular framework for hallucination detection in video question answering that extends entropy-based reliability estimation from images to temporally structured inputs. Given a video-question pair, VideoHEDGE draws a baseline answer and multiple high-temperature generations from both clean clips and photometrically and spatiotemporally perturbed variants, then clusters the resulting textual outputs into semantic hypotheses using either Natural Language Inference (NLI)-based or embedding-based methods. Cluster-level probability masses yield three reliability scores: Semantic Entropy (SE), RadFlag, and Vision-Amplified Semantic Entropy (VASE). We evaluate VideoHEDGE on the SoccerChat benchmark using an LLM-as-a-judge to obtain binary hallucination labels. Across three 7B Video-VLMs (Qwen2-VL, Qwen2.5-VL, and a SoccerChat-finetuned model), VASE consistently achieves the highest ROC-AUC, especially at larger distortion budgets, while SE and RadFlag often operate near chance. We further show that embedding-based clustering matches NLI-based clustering in detection performance at substantially lower computational cost, and that domain fine-tuning reduces hallucination frequency but yields only modest improvements in calibration. The hedge-bench PyPI library enables reproducible and extensible benchmarking, with full code and experimental resources available at https://github.com/Simula/HEDGE#videohedge .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。