通过动态调整解码策略，减少大模型视觉问答中的幻觉问题。
Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models
- 根据注意力一致性判断是否正确关注图像，动态选择解码方式。
- 在多个主流数据集上显著降低幻觉率，优于现有方法。
- 适合需要高可信度生成结果的视觉语言任务应用。
大型视觉语言模型（LVLMs）在各类视觉任务中展现出强大能力，但仍受幻觉问题困扰。为此，我们提出一种名为混合解码（Mixture of Decoding, MoD）的新方法，通过评估模型对图像标记的注意力正确性，动态自适应调整解码策略。具体而言，MoD测量原始图像标记与模型注意力所聚焦图像标记生成输出的一致性，以判断注意力是否正确。若输出一致，表明注意力准确，采用互补策略增强关键信息；若不一致，说明注意力错误，采用对比策略抑制误导信息。大量实验表明，MoD在多个主流基准上显著优于现有解码方法，有效缓解了LVLM中的幻觉现象。代码已开源：https://github.com/xlchen0205/MoD。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) have exhibited impressive capabilities across various visual tasks, yet they remain hindered by the persistent challenge of hallucinations. To address this critical issue, we propose Mixture of Decoding (MoD), a novel approach for hallucination mitigation that dynamically adapts decoding strategies by evaluating the correctness of the model's attention on image tokens. Specifically, MoD measures the consistency between outputs generated from the original image tokens and those derived from the model's attended image tokens, to distinguish the correctness aforementioned. If the outputs are consistent, indicating correct attention, MoD employs a complementary strategy to amplify critical information. Conversely, if the outputs are inconsistent, suggesting erroneous attention, MoD utilizes a contrastive strategy to suppress misleading information. Extensive experiments demonstrate that MoD significantly outperforms existing decoding methods across multiple mainstream benchmarks, effectively mitigating hallucinations in LVLMs. The code is available at https://github.com/xlchen0205/MoD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。