提出无需训练的SHIELD框架，抑制视觉编码器引发的幻觉
SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defense
- 通过重加权、噪声令牌和对抗攻击缓解三类偏差
- 在多个基准上显著降低物体幻觉率，提升模型可靠性
- 适用于多种视觉语言模型，无需额外训练
大型视觉-语言模型（LVLM）在跨模态任务中表现优异，但物体幻觉问题仍严重：模型生成看似合理却错误的物体描述。不同于以往聚焦大语言模型的研究，本文首次将幻觉根源定位到视觉编码器，识别出三类核心问题：统计偏差、固有偏差和脆弱性。为此，我们提出SHIELD——一种无需训练的防御框架，采用三种策略：重加权视觉标记以减少统计偏差，引入噪声衍生标记以对抗固有偏差，以及通过对比解码实施对抗攻击来应对脆弱性。实验表明，SHIELD在多个基准和不同LVLM架构中均有效缓解物体幻觉，且在通用LVLM基准上表现优异，展现出广泛适用性。代码已公开于https://github.com/hukcc/SHIELD。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) excel in diverse cross-modal tasks. However, object hallucination, where models produce plausible but inaccurate object descriptions, remains a significant challenge. In contrast to previous work focusing on LLM components, this paper is the first to trace LVLM hallucinations to visual encoders and identifies three key issues: statistical bias, inherent bias, and vulnerability. To address these challenges, we propose SHIELD, a training-free framework that mitigates hallucinations through three strategies: re-weighting visual tokens to reduce statistical bias, introducing noise-derived tokens to counter inherent bias, and applying adversarial attacks with contrastive decoding to address vulnerability. Experiments demonstrate that SHIELD effectively mitigates object hallucinations across diverse benchmarks and LVLM families. Moreover, SHIELD achieves strong performance on the general LVLM benchmark, highlighting its broad applicability. Code is available at https://github.com/hukcc/SHIELD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。