无需训练,用生成图像做锚点修复视觉语言模型的幻觉问题
Exposing Hallucinations To Suppress Them: VLMs Representation Editing With Generative Anchors
- 用文本生成图像作为负面锚点,原图作正锚点,编辑模型隐状态
- 在CHAIR数据集上减少超5%幻觉,同时保持回复丰富性和召回率
- 方法无需额外训练或人工规则,可直接接入多种主流模型
多模态大语言模型在视觉-语言任务中表现优异,但仍易产生与视觉证据不符的幻觉内容。现有缓解方法常需微调、手工先验或牺牲信息量和可扩展性。为此,我们提出一种无需训练的自监督幻觉抑制方法:通过文本生成图像揭示隐含幻觉信号,作为负锚点,原图作为正锚点,利用双锚点编辑解码器隐藏状态,使表征向真实语义靠拢、远离幻觉方向。该方法无需人工先验或额外训练成本,兼具有效性和效率。在多个基准测试中,该方法显著降低物体、属性和关系层面的幻觉,如在LLaVA-v1.5-7B上于CHAIR数据集实现超过5%的幻觉减少,同时保持高召回率和描述丰富性。在LLaVA-NEXT-7B、Cambrian-8B、InstructBLIP-7B等不同架构上均表现出强泛化能力。应用于无幻觉标题时几乎无副作用,体现其鲁棒性与即插即用特性。代码将公开。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have achieved remarkable success across diverse vision-language tasks, yet they remain highly susceptible to hallucinations, producing content that is fluent but inconsistent with visual evidence. Such hallucinations, spanning objects, attributes, and relations, persist even in larger models, while existing mitigation approaches often require additional finetuning, handcrafted priors, or trade-offs that compromise informativeness and scalability. To address this limitation, we propose a training-free, self-supervised method for hallucination mitigation. Our approach introduces a novel hallucination amplification mechanism: a caption is projected into the visual space via a text-to-image model to reveal implicit hallucination signals, serving as a negative anchor, while the original image provides a positive anchor. Leveraging these dual anchors, we edit decoder hidden states by pulling representations toward faithful semantics and pushing them away from hallucination directions. This correction requires no human priors or additional training costs, ensuring both effectiveness and efficiency. Extensive experiments across multiple benchmarks show that our method significantly reduces hallucinations at the object, attribute, and relation levels while largely preserving recall and caption richness, e.g., achieving a hallucination reduction by over 5% using LLaVA-v1.5-7B on CHAIR. Furthermore, results on diverse architectures, including LLaVA-NEXT-7B, Cambrian-8B, and InstructBLIP-7B, validate strong cross-architecture generalization. More importantly, when applied to hallucination-free captions, our method introduces almost no side effects, underscoring its robustness and practical plug-and-play applicability. The implementation will be publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。