提出BICR方法,让大模型自检视觉回答是否真依赖图像。
Grounded or Guessing? LVLM Confidence Estimation via Blind-Image Contrastive Ranking

- 用遮蔽图像对比真实图像的隐藏状态,显式判断模型是否真看图
- 在5个模型上同时提升校准与判别性能,效果优于所有基线
- 只需少量参数(4-18倍更少),即可检测视觉幻觉
大型视觉语言模型存在视觉不落地问题:模型可仅凭语言先验生成流畅且自信的回答,图像对预测无实际贡献。现有置信度估计方法无法识别此类情况,因其在正常推理下观察模型行为,缺乏机制判断预测是否由图像驱动。本文提出BICR(盲图对比排序)框架,通过两次提取冻结的LVLM隐藏状态实现显式对比:一次使用真实图像-问题对,一次在图像被遮蔽而问题保持不变时进行。训练一个轻量级探测器,利用真实图像的隐藏状态,并通过排名损失惩罚在遮蔽视图中更高的置信度,从而教会探测器将视觉接地性视为可靠性的信号,且无需额外推理开销。在涵盖视觉问答、物体幻觉检测、医学影像和金融文档理解的基准上,评估了五种现代LVLM和七种基线,BICR在跨模型平均校准与判别性能上均达最优,判别性能提升具有统计显著性,且在聚类感知分析中表现稳健,参数量仅为最强探测基线的4-18倍。
原文摘要 · Abstract (English)
Large vision-language models suffer from visual ungroundedness: they can produce a fluent, confident, and even correct response driven entirely by language priors, with the image contributing nothing to the prediction. Existing confidence estimation methods cannot detect this, as they observe model behavior under normal inference with no mechanism to determine whether a prediction was shaped by the image or by text alone. We introduce BICR (Blind-Image Contrastive Ranking), a model-agnostic confidence estimation framework that makes this contrast explicit during training by extracting hidden states from a frozen LVLM twice: once with the real image-question pair, and once with the image blacked out while the question is held fixed. A lightweight probe is trained on the real-image hidden state and regularized by a ranking loss that penalizes higher confidence on the blacked-out view, teaching it to treat visual grounding as a signal of reliability at zero additional inference cost. Evaluated across five modern LVLMs and seven baselines on a benchmark covering visual question answering, object hallucination detection, medical imaging, and financial document understanding, BICR achieves the best cross-LVLM average on both calibration and discrimination simultaneously, with statistically significant discrimination gains robust to cluster-aware analysis at 4-18x fewer parameters than the strongest probing baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。