arXiv:2601.05547cs.CVcs.AI2026-01ACL被引 10

通过信息瓶颈原理,精准定位并抑制视觉语言模型的幻觉生成。

VIB-Probe: Detecting and Mitigating Hallucinations in Vision-Language Models via Variational Information Bottleneck

  • 利用变分信息瓶颈提取注意力头中的关键语义模式。
  • 在多个基准上检测幻觉准确率显著优于现有方法。
  • 适合关注模型可信性与可解释性的研究者使用。

视觉语言模型(VLMs)在多模态任务中表现卓越,但仍易产生幻觉,即生成内容偏离图像真实语义。现有检测方法多依赖输出概率或外部验证工具,忽视内部机制。本文探究内部注意力头的输出,提出假设:特定注意力头携带真实生成的主要信号。然而,高维状态中的视觉-语言语法与噪声纠缠,使直接探测困难。为此,我们提出VIB-Probe框架,基于变分信息瓶颈(VIB)理论,提取跨层、跨头的判别性模式,同时通过信息瓶颈原则过滤语义干扰。此外,利用VIB探针的梯度,识别对幻觉具有强因果影响的注意力头,并设计推理时干预策略以减轻幻觉。大量实验表明,该方法在多种基准上均显著优于现有基线。代码将公开。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) have demonstrated remarkable progress in multimodal tasks, but remain susceptible to hallucinations, where generated text deviates from the underlying visual content. Existing hallucination detection methods primarily rely on output logits or external verification tools, often overlooking their internal mechanisms. In this work, we investigate the outputs of internal attention heads, postulating that specific heads carry the primary signals for truthful generation.However, directly probing these high-dimensional states is challenging due to the entanglement of visual-linguistic syntax and noise. To address this, we propose VIB-Probe, a novel hallucination detection and mitigation framework leveraging the Variational Information Bottleneck (VIB) theory. Our method extracts discriminative patterns across layers and heads while filtering out semantic nuisances through the information bottleneck principle. Furthermore, by leveraging the gradients of our VIB probe, we identify attention heads with strong causal influence on hallucinations and introduce an inference-time intervention strategy for hallucination mitigation. Extensive experiments across diverse benchmarks demonstrate that VIB-Probe significantly outperforms existing baselines in both settings. Our code will be made publicly available.

幻觉检测视觉语言模型信息瓶颈

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。