通过对比正负路径干预解码,减少视觉语言模型的幻觉生成。
Global Context or Local Detail? Adaptive Visual Grounding for Hallucination Mitigation

- 双路径对比机制:正向强化视觉特征,负向抑制核心对象特征
- 在POPE等数据集上提升6.5%准确率,显著降低幻觉率
- 无需重训练,适配LLaVA、Qwen-VL等多种主流模型
视觉语言模型(VLMs)常因过度依赖语言先验而产生与视觉事实不符的对象幻觉。本文提出无需训练的推理框架PND,直接干预解码过程以保障视觉真实性。其核心发现是VLM存在注意力缺陷——视觉特征被低估。PND通过双路径对比机制解决:正路径利用多层注意力放大显著视觉证据,促进忠实描述;负路径识别并削弱核心对象特征,生成强反事实,惩罚依赖先验的生成。在每一步对比两个视角下的输出,引导模型生成不仅语言合理,且视觉真实的文本。在POPE、MME和CHAIR等基准上实验表明,PND实现最先进性能,最高提升6.5%准确率,大幅减少幻觉同时增强描述细节,且不需模型重训练,可泛化至LLaVA、InstructBLIP、InternVL和Qwen-VL等多种架构。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) are frequently undermined by object hallucination--generating content that contradicts visual reality--due to an over-reliance on linguistic priors. We introduce Positive-and-Negative Decoding (PND), a training-free inference framework that intervenes directly in the decoding process to enforce visual fidelity. PND is motivated by our key finding of a critical attention deficit in VLMs, where visual features are empirically under-weighted. Our framework corrects this via a dual-path contrast: The positive path amplifies salient visual evidence using multi-layer attention to encourage faithful descriptions, directly counteracting the attention deficit. Simultaneously, the negative path identifies and degrades the core object's features to create a strong counterfactual, which penalizes ungrounded, prior-dominant generation. By contrasting the model's outputs from these two perspectives at each step, PND steers generation towards text that is not just linguistically probable, but visually factual. Extensive experiments on benchmarks like POPE, MME, and CHAIR show that PND achieves state-of-the-art performance with up to 6.5% accuracy improvement, substantially reducing object hallucination while also enhancing descriptive detail--all without requiring any model retraining. The method generalizes effectively across diverse VLM architectures including LLaVA, InstructBLIP, InternVL, and Qwen-VL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。