用能量引导解码，减少视觉语言模型的错误生成
Energy-Guided Decoding for Object Hallucination Mitigation
- 基于能量最小化选择隐藏状态，动态优化解码过程
- 平均准确率提升4.82%，'是'的回答偏差降低8.81%
- 无需外部知识或输入修改，适配主流视觉语言模型
大型视觉语言模型（LVLMs）中的对象幻觉问题对其安全部署至关重要。现有方法或仅适用于特定解码方式，或需复杂修改视觉输入，或依赖外部模型知识。本文首次揭示，在三个不同视觉问答（VQA）数据集上，LVLMs存在显著的'是'回答比例失衡现象。为此，我们提出一种基于能量的解码方法，动态选取能量得分最低层的隐藏状态。该方法简单有效，在三个基准测试（POPE、MME、MMVP）上均能降低'是'回答偏差并提升性能。相比贪婪解码，平均准确率提升4.82%，平均'是'比例差距减少8.81%，表明该方法更少偏倚，且在三种常用LVLM上均优于多个基线方法。
原文摘要 · Abstract (English)
Mitigating object hallucination in large vision-language models (LVLMs) is critical to their safe deployment. Existing methods either are restricted to specific decoding methods, or demand sophisticated modifications to visual inputs, or rely on knowledge from external models. In this work, we first reveal the phenomenon that VLMs exhibit significant imbalance in the ``Yes'' ratio ( \ie, the fraction of ``Yes'' answers among the total number of questions) across three different visual question answering (VQA) datasets. Furthermore, we propose an energy-based decoding method, which dynamically selects the hidden states from the layer with minimal energy score. It is simple yet effective in reducing the bias for the yes ratio while boosting performance across three benchmarks (POPE, MME, and MMVP). Our method consistently improves accuracy and F1 score on three VQA datasets across three commonly used VLMs over several baseline methods. The average accuracy improvement is 4.82% compared to greedy decoding. Moreover, the average yes-ratio gap reduction is 8.81%, meaning the proposed method is less biased as shown in Figure 1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。