通过动态检测并编辑幻觉表示，高效消除视觉语言模型的幻觉问题。
Hallucination-aware intermediate representation edit in large vision-language models
- 动态识别幻觉相关表示并针对性编辑
- 在多个基准上达到顶尖消幻觉效果
- 计算开销小，适合实际部署
大型视觉语言模型在多模态推理和复杂场景理解方面表现出色，但仍存在严重幻觉问题，即输出与视觉事实矛盾。现有幻觉缓解方法主要分为微调和对比解码（CD）两类。前者需大量训练资源，后者带来双重推理开销，限制了实际应用。为此，我们提出一种框架，可动态检测幻觉表示，并对这些表示进行消除幻觉的编辑。该方法仅引入极小额外计算成本，便在现有基准上实现了最先进的性能。大量实验证明其在幻觉消除上的高效性、鲁棒性及对幻觉的强大可控性。代码已公开于 https://github.com/ASGO-MM/HIRE。
原文摘要 · Abstract (English)
Large Vision-Language Models have demonstrated exceptional performance in multimodal reasoning and complex scene understanding. However, these models still face significant hallucination issues, where outputs contradict visual facts. Recent research on hallucination mitigation has focused on retraining methods and Contrastive Decoding (CD) methods. While both methods perform well, retraining methods require substantial training resources, and CD methods introduce dual inference overhead. These factors hinder their practical applicability. To address the above issue, we propose a framework for dynamically detecting hallucination representations and performing hallucination-eliminating edits on these representations. With minimal additional computational cost, we achieve state-of-the-art performance on existing benchmarks. Extensive experiments demonstrate the effectiveness of our approach, highlighting its efficient and robust hallucination elimination capability and its powerful controllability over hallucinations. Code is available at https://github.com/ASGO-MM/HIRE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。