通过引导视觉标记增强语言生成，减少大模型幻觉。
Revisit What You See: Revealing Visual Semantics in Vision Tokens to Guide LVLM Decoding
- 不需训练，用视觉标记语义动态修正文本生成分布。
- 在五个基准上超越或媲美现有方法，计算成本降低两倍。
- 适合希望提升视觉理解准确性且优化推理效率的研究者。
大型视觉语言模型（LVLMs）通过融合视觉感知与语言理解，在多模态任务中表现优异。然而，视觉信息如何影响模型解码过程仍不清楚，导致频繁出现幻觉。我们通过一系列分析发现：(i) 即使发生幻觉，视觉标记仍包含有意义的视觉信息；(ii) 其语义被编码在文本空间中，并在适当词汇约束下变得明确。基于此，我们提出 ReVisiT——一种无需训练的解码方法，通过参考视觉标记来引导文本生成。该方法将视觉标记投影至文本词元分布，通过上下文感知的约束分歧最小化，动态选择最相关视觉标记，并利用其约束投影优化输出分布，以更好地融入视觉语义。在多个近期 LVLM 的五个基准测试中，ReVisiT 达到与当前最优解码基线相当或更优的表现，同时计算成本降低高达 2 倍。
原文摘要 · Abstract (English)
Large Vision Language Models (LVLMs) achieve strong performance across multimodal tasks by integrating visual perception with language understanding. However, how vision information contributes to the model's decoding process remains under-explored, as reflected in frequent hallucinations. Through a series of analyses, we found that (i) vision tokens provide meaningful visual information even when hallucinations occur, and (ii) their semantics are encoded in the textual space and become explicit under appropriate vocabulary constraints. Building on these observations, we propose ReVisiT, a simple training-free decoding method that guides text generation in LVLMs by Referencing Vision Tokens. Our approach leverages the semantic information embedded within vision tokens by projecting them into the text token distribution. Specifically, ReVisiT dynamically selects the most relevant vision token at each decoding step via context-aware constrained divergence minimization. Then, ReVisiT uses its constrained projection to refine the output distribution to better incorporate visual semantics. Across five benchmarks on recent LVLMs, ReVisiT achieves competitive or superior results to state-of-the-art decoding baselines while reducing computational cost by up to $2\times$
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。