arXiv:2505.17529cs.CVcs.AI2025-05ICLR被引 13

用注意力加权融合子图推理,有效减少视觉语言模型的幻觉错误。

Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decoding

  • 将图像拆分为子图,通过注意力图加权融合各部分的输出结果。
  • 在多个基准上达到当前最优性能,显著降低虚构物体生成率。
  • 适合需要高准确率且对速度有要求的应用场景。

大型视觉语言模型(LVLMs)在图像描述和视觉问答等任务中取得显著进展,但仍存在对象幻觉问题,即模型会生成与视觉内容不符的描述,包括虚构不存在的对象或错误描述已有对象。尽管先前方法如数据增强和无需训练的策略试图缓解此问题,仍面临可扩展性挑战,并常依赖额外模块。本文提出集成解码(Ensemble Decoding, ED),将输入图像分割为子图像,利用注意力图为各子图像的词元分布分配权重并进行融合。进一步引入自适应合理性约束以校准词元分布,并设计快速版FastED,适用于对速度敏感的应用。在多个幻觉评估基准上的实验表明,所提方法达到当前最优性能,验证了其有效性。

原文摘要 · Abstract (English)

Recent advancements in Large Vision-Language Models (LVLMs) have significantly expanded their utility in tasks like image captioning and visual question answering. However, they still struggle with object hallucination, where models generate descriptions that inaccurately reflect the visual content by including nonexistent objects or misrepresenting existing ones. While previous methods, such as data augmentation and training-free approaches, strive to tackle this issue, they still encounter scalability challenges and often depend on additional external modules. In this work, we propose Ensemble Decoding (ED), a novel strategy that splits the input image into sub-images and combines logit distributions by assigning weights through the attention map. Furthermore, we introduce ED adaptive plausibility constraint to calibrate logit distribution and FastED, a variant designed for speed-critical applications. Extensive experiments across hallucination benchmarks demonstrate that our proposed method achieves state-of-the-art performance, validating the effectiveness of our approach.

视觉语言模型幻觉抑制注意力机制集成解码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。