无需训练，一次解码消除视觉幻觉。
RVSD: Retrieval Vision Sparse Decoding for Mitigating Visual Hallucinations in Large Vision-Language Models
- 通过语义引导选择关键视觉令牌，稀疏冗余信息。
- 在共享语义空间中按需检索视觉内容，实时补偿。
- 适合需要高可靠性的视觉语言模型应用者。
大型视觉-语言模型在视觉-语言任务中取得了显著进展，但仍易产生视觉幻觉（VH），影响其在真实场景中的可靠性。现有方法通常需要精心构建的数据集、额外训练或多轮解码，带来较大计算开销。本文提出 extbf{RVSD}（检索视觉稀疏解码），一种无需训练、即插即用的解码框架，首次在单次解码过程中统一了令牌稀疏化与 extbf{语义空间视觉检索}（SSVR）。RVSD引入语义导向的令牌选择策略，选择性稀疏化冗余令牌，同时保留关键视觉信息。进一步提出SSVR机制，将视觉补偿重构为共享语义空间内的按需跨模态检索。大量实验表明，RVSD在抑制视觉幻觉方面达到当前最优性能，并在长上下文生成设置下保持强鲁棒性。代码已开源。
原文摘要 · Abstract (English)
Large vision-language models have achieved remarkable success in vision-language tasks. However, they remain prone to Visual Hallucinations (VHs), undermining their reliability in real-world applications. Existing solutions typically require curated datasets, additional training, or multi-round decoding, resulting in considerable computational overhead. In this paper, we propose \textbf{RVSD} (\underline{R}etrieval \underline{V}ision \underline{S}parse \underline{D}ecoding), a training-free and plug-and-play decoding framework that, for the first time, unifies token sparsification and \textbf{Semantic-Space Visual Retrieval} (SSVR) within a single decoding pass. Within RVSD, we introduce a \textbf{semantics-directed token selection} strategy that selectively sparsifies redundant tokens while preserving critical visual information. We further propose the SSVR mechanism, which reformulates visual compensation as an on-demand cross-modal retrieval process within a shared semantic space. Extensive experiments demonstrate that RVSD achieves state-of-the-art performance in mitigating VHs while maintaining robust suppression capabilities under long-context generation settings. Our code is available here.\footnote{https://github.com/canjie-liu/RVSD}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。