提出PATCH方法，通过虚拟令牌减少视觉语言模型的幻觉问题。
From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models
- 用自适应虚拟令牌从框中提取对象特征，改进特征解耦。
- 在多个多模态幻觉数据集上达到顶尖性能。
- 适合研究视觉语言模型幻觉机制与鲁棒性提升的学者。
大型视觉语言模型（LVLM）中的幻觉问题严重影响其可靠性，即生成视觉输入中不存在的对象。现有研究多归因于对视觉输入理解不足，却忽略了更根本的问题：模型无法有效提取或解耦视觉特征。本文从架构视角重新审视幻觉成因，探究问题根源是否在于视觉编码器（特征提取）或模态对齐模块（特征解耦）。基于初步发现，我们提出一种新微调策略PATCH，可无缝集成于多种LVLM中。该方法利用自适应虚拟令牌，从边界框中提取对象特征，从而缓解由特征解耦不足引发的幻觉。PATCH在多个多模态幻觉数据集上取得当前最优表现。我们希望此工作为深入理解LVLM幻觉根源提供新视角，推动该领域进一步发展。
原文摘要 · Abstract (English)
Hallucinations in large vision-language models (LVLMs) are a significant challenge, i.e., generating objects that are not presented in the visual input, which impairs their reliability. Recent studies often attribute hallucinations to a lack of understanding of visual input, yet ignore a more fundamental issue: the model's inability to effectively extract or decouple visual features. In this paper, we revisit the hallucinations in LVLMs from an architectural perspective, investigating whether the primary cause lies in the visual encoder (feature extraction) or the modal alignment module (feature decoupling). Motivated by our findings on the preliminary investigation, we propose a novel tuning strategy, PATCH, to mitigate hallucinations in LVLMs. This plug-and-play method can be integrated into various LVLMs, utilizing adaptive virtual tokens to extract object features from bounding boxes, thereby addressing hallucinations caused by insufficient decoupling of visual features. PATCH achieves state-of-the-art performance on multiple multi-modal hallucination datasets. We hope this approach provides researchers with deeper insights into the underlying causes of hallucinations in LVLMs, fostering further advancements and innovation in this field.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。