让视觉语言模型自我纠错，减少幻觉生成。
Toward More Reliable Artificial Intelligence: Reducing Hallucinations in Vision-Language Models
- 通过不确定性引导重关注，迭代修正错误回答。
- 在对抗性数据集上幻觉率降低9.8%，物体存在准确率提升4.7%。
- 无需训练，适配主流模型，提升AI可信度。
视觉语言模型（VLMs）常生成看似合理却与图像内容不符的幻觉内容。本文提出一种无需训练的自校正框架，使VLM通过不确定性引导的视觉重关注机制，迭代优化回答。该方法结合多维度不确定性量化（词元熵、注意力分散度、语义一致性、声明置信度）与注意力引导的区域裁剪，仅使用冻结的预训练模型，无需梯度更新。我们在POPE和MMHAL BENCH基准上以Qwen2.5-VL-7B [23]架构验证该方法，结果表明，相比基线，幻觉率降低9.8个百分点，对抗性数据集上的物体存在准确率提升4.7个百分点。定性分析显示，不确定性引导的重关注能有效将修正基于视觉证据，而标准解码无法做到。未来版本将扩展至多种模型架构。代码与方法已公开，助力可信赖多模态系统研究。
原文摘要 · Abstract (English)
Vision-language models (VLMs) frequently generate hallucinated content plausible but incorrect claims about image content. We propose a training-free self-correction framework enabling VLMs to iteratively refine responses through uncertainty-guided visual re-attention. Our method combines multidimensional uncertainty quantification (token entropy, attention dispersion, semantic consistency, claim confidence) with attention-guided cropping of under-explored regions. Operating entirely with frozen, pretrained VLMs, our framework requires no gradient updates. We validate our approach on the POPE and MMHAL BENCH benchmarks using the Qwen2.5-VL-7B [23] architecture. Experimental results demonstrate that our method reduces hallucination rates by 9.8 percentage points compared to the baseline, while improving object existence accuracy by 4.7 points on adversarial splits. Furthermore, qualitative analysis confirms that uncertainty-guided re-attention successfully grounds corrections in visual evidence where standard decoding fails. We validate our approach on Qwen2.5-VL-7B [23], with plans to extend validation across diverse architectures in future versions. We release our code and methodology to facilitate future research in trustworthy multimodal systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。