用轻量模块增强视觉记忆，减少视觉语言模型幻觉
ReCo: Reminder Composition Mitigates Hallucinations in Vision-Language Models
- 引入可训练模块ReCo，通过关系组合强化视觉记忆
- 在InstructBLIP、LlaVA等模型上显著降低幻觉率
- 兼容多种去幻觉方法，适合提升现有VLM可靠性
视觉语言模型（VLMs）在融合与推理视觉和语言数据方面表现出色，但常出现幻觉——生成看似合理却与视觉输入不符甚至矛盾的内容。研究表明，这源于模型对语言的过度依赖，尤其在生成过程中出现‘记忆消退’现象。本文提出一种轻量级可训练模块ReCo，基于几何代数与关系组合思想，可无缝集成于任意VLM之上，无需其他修改。实验表明，该模块在三个主流VLM（InstructBLIP、LlaVA、MiniGPT4）上均有效缓解记忆消退问题，在多个基准测试中提升性能。此外，ReCo可与多种现有去幻觉方法结合，进一步改善效果。
原文摘要 · Abstract (English)
Vision Language Models (VLMs) show impressive capabilities in integrating and reasoning with both visual and language data. But these models make mistakes. A common finding -- similar to LLMs -- is their tendency to hallucinate, i.e., generate plausible sounding text which is not grounded in the visual input, or at worst, is contradictory. A growing consensus attributes this behavior to an over-reliance on language -- especially as the generation progresses, the model suffers from a ``fading memory effect'' with respect to the provided visual input. We study mechanisms by which this behavior can be controlled. Specifically, using ideas from geometric algebra and relational compositions, we propose the addition of a small, trainable module (named ReCo) on top of any VLM -- no other modification is needed. We show that such a lightweight module is able to mitigate the fading memory effect on three of the most widely used VLMs (InstructBLIP, LlaVA, MiniGPT4), where we see performance improvements on multiple benchmarks. Additionally, we show that our module can be combined with many of the other approaches for reducing hallucination where we achieve improved results for each one.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。