用视觉信息提升图文生成模型的不确定性估计精度
Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generation
- 通过视觉接地分数加权语言模态不确定度
- 在多个模型架构上均提升预测可靠性
- 无需训练，适用于各类图文生成系统
不确定性量化（UQ）是大型视觉语言模型（LVLMs）实现可靠预测与实际部署的关键挑战。现有方法多源自大语言模型（LLM）文献，主要关注语言模态，忽视了视觉信息对LVLM不确定性的影响。本文研究了LVLM如何处理视觉信息，并探索其在不确定性估计中的作用。分析生成过程中视觉特征融合后的隐藏表示发现，高置信度预测更依赖视觉内容，而低置信度预测则相反。基于此，提出无需训练的视觉锚定令牌级不确定性估计框架（VIG-TUQ），通过将视觉接地得分与语言模态不确定性加权结合，增强不确定性判断。在多个数据集及多种架构（早融合、晚融合、原生融合）上验证，结果表明该方法显著优于现有方法。代码与数据将在录用后公开。
原文摘要 · Abstract (English)
Uncertainty quantification (UQ) remains a critical challenge in Large Vision Language Models (LVLMs) for reliable predictions and real-world deployment. However, most existing methods are adapted from the LLM literature and primarily focus on the language modality, leaving the contribution of visual information to LVLM uncertainty largely underexplored. In this paper, we investigate how LVLMs process visual information and whether this process can be used to improve uncertainty estimation. By analyzing hidden representations after the integration of visual features during the generation process, we observe that high-confidence predictions rely more heavily on visual content than uncertain ones. Building on this insight, we propose Visual-Grounded Token UQ (VIG-TUQ), a training-free framework that explicitly incorporates visual grounding into uncertainty estimation by weighting token-level language uncertainty with visual grounding scores. We evaluate VIG-TUQ on multiple datasets and across diverse LVLM architectures, including early-fusion, late-fusion, and native-fusion models. Results indicate that our method often improves upon existing token-level uncertainty approaches. Code and data will be made available upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。