提出无训练方法，双模态抑制视觉语言模型幻觉。
Watch Closely: Mitigating Object Hallucinations in Large Vision-Language Models with Disentangled Decoding
- 分离解码：用分割图和空白图减少语言先验依赖。
- 在COCO、VQAv2上物体识别准确率提升3.2%~5.1%。
- 适合需高可信生成的医疗、自动驾驶场景。
大型视觉语言模型（LVLM）在跨模态任务中展现出强大潜力，但物体识别时仍存在严重幻觉问题。模型常无法准确识别某些物体，导致生成文本看似流畅却与图像内容不符，影响实际应用。现有方法多仅缓解语言模态幻觉，本研究提出无需训练的幻觉解耦解码（HDD）方法：通过图像分割生成增强图，并引入空白图消除原图与分割图中的语言先验幻觉。该设计降低模型对语言先验的依赖，同时提升视觉表现力，在COCO和VQAv2数据集上平均提升3.2%~5.1%的物体识别准确率。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) bridge the gap between visual and linguistic modalities, demonstrating strong potential across a variety of domains. However, despite significant progress, LVLMs still suffer from severe hallucination issues in object recognition tasks. These models often fail to accurately identify certain objects, leading to text generation that appears fluent but does not correspond to the visual content, which can have serious consequences in real-world applications. Recently, several methods have been proposed to alleviate LVLM hallucinations, but most focus solely on reducing hallucinations in the language modality. To mitigate hallucinations in both the language and visual modalities, we introduce Hallucination Disentangled Decoding (HDD) method that requires no training. HDD enhances the original image by segmenting it and selecting images that augment the original, while also utilizing a blank image to eliminate language prior hallucinations in both the original and segmented images. This design not only reduces the model's dependence on language priors but also enhances its visual performance. (Code: https://github.com/rickeyhhh/Hallucination-Disentangled-Decoding)
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。