动态对比解码框架，自适应抑制视觉语言模型幻觉
Octopus: Alleviating Hallucination via Dynamic Contrastive Decoding
- 根据生成阶段动态识别幻觉类型，调整对比解码策略
- 在四个基准上优于现有方法，显著降低幻觉率
- 结构简单易部署，适合实际应用与扩展
大型视觉语言模型在视觉理解与多模态推理方面表现优异，但普遍存在严重幻觉问题，倾向于生成虚构内容。近期提出的对比解码（CD）策略通过引入扰动输入缓解幻觉，但大多采用固定策略，未考虑不同输入条件下的差异。本文通过大量实验发现，幻觉成因复杂，每个生成步骤面临独特的幻觉挑战。基于此，我们提出类章鱼的动态对比解码框架（Octopus），使模型能自适应识别幻觉类型并构建动态解码流程。该框架在四个基准测试中均超越现有方法，且具备良好可部署性与可扩展性。代码已公开于 https://github.com/LijunZhang01/Octopus。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) have obtained impressive performance in visual content understanding and multi-modal reasoning. Unfortunately, these large models suffer from serious hallucination problems and tend to generate fabricated responses. Recently, several Contrastive Decoding (CD) strategies have been proposed to alleviate hallucination by introducing disturbed inputs. Although great progress has been made, these CD strategies mostly apply a one-size-fits-all approach for all input conditions. In this paper, we revisit this process through extensive experiments. Related results show that hallucination causes are hybrid and each generative step faces a unique hallucination challenge. Leveraging these meaningful insights, we introduce a simple yet effective Octopus-like framework that enables the model to adaptively identify hallucination types and create a dynamic CD workflow. Our Octopus framework not only outperforms existing methods across four benchmarks but also demonstrates excellent deployability and expansibility. Code is available at https://github.com/LijunZhang01/Octopus.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。