通过双分支重平衡注意力，减少视觉语言模型的幻觉问题。
Mitigating Hallucination in Visual-Language Models via Re-Balancing Contrastive Decoding
- 用文本和视觉双分支重新校准模型注意力分布。
- 在CHAIR和POPE指标上优于现有方法，有效减少幻觉。
- 适合关注多模态模型可靠性与真实性的研究者。
尽管视觉语言模型（VLMs）在图像问答和图像描述等任务中表现优异，但仍存在幻觉问题。分析显示，这类模型倾向于过度关注文本而非视觉信息，导致在多模态知识冲突时更依赖文本，与图像内容不符。本文提出重平衡对比解码（RBD）方法，通过文本分支注入图像噪声以削弱对文本的依赖，同时视觉分支聚焦关键标记选择，优化注意力机制以突出主体。该双分支策略在降低文本偏见的同时增强视觉信息利用。实验表明，RBD在CHAIR和POPE指标上超越现有方法，有效缓解幻觉且不损害模型通用能力。
原文摘要 · Abstract (English)
Although Visual-Language Models (VLMs) have shown impressive capabilities in tasks like visual question answering and image captioning, they still struggle with hallucinations. Analysis of attention distribution in these models shows that VLMs tend to processing textual tokens rather than visual tokens. This imbalance of attention distribution causes VLMs to favor textual knowledge in the case of multimodal knowledge conflicts, resulting in differences from the image information. In this paper, we propose Re-Balancing Contrastive Decoding (RBD) method, which employs textual and visual branches to recalibrate attention distribution in VLMs. Specifically, the textual branch injects image noise to stimulate the model's dependency on text, thereby reducing textual bias. Concurrently, the visual branch focuses on the selection of significant tokens, refining the attention mechanism to highlight the primary subject. This dual-branch strategy enables the RBD method to diminish textual bias while enhancing visual information. Experimental results demonstrate that our method, RBD, outperforms the existing methods by the CHAIR and POPE metrics, mitigate hallucinations without reducing the model's general capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。