通过视觉回溯评分提升大模型回答准确性
LookBack: Where and How to Score LVLM Responses via Visual Reference Usage
- 用视觉回溯分数衡量每个词是否对应图像内容
- 在四个基准上优于现有方法，且开销极小
- 适合需要高可信度图文生成的场景
大型视觉语言模型（LVLM）融合视觉感知与语言生成，可实现跨图像理解与复杂推理。然而，这类模型不仅继承文本幻觉，还会产生违背图像内容的流畅回答，导致评分难度上升。我们的诊断表明，现有基于置信度的评估指标（源自大语言模型）对LVLM不适用：移除输入图像后，置信度选择基本不变，说明输出空间置信度主要反映文本合理性而非与图像的一致性。为此，我们提出LookBack——一种无需训练的响应评分方法，通过在词元似然基础上增加视觉回溯分数（visual lookback score），量化每个回答词元与图像词元的关联强度。在四个基准和三种模型上，LookBack均显著提升Best-of-$N$选择性能，且计算开销可忽略。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) integrate visual perception with language generation, enabling responses that span image understanding and complex reasoning. However, LVLMs do not just inherit the text-level hallucinations; they also hallucinate against the image, producing fluent responses ungrounded in what they see. This makes LVLM response scoring inherently harder, and our diagnostics show that existing confidence-based metrics adopted from LLMs are insufficient for LVLMs. Specifically, removing the input image barely changes confidence-based selection, suggesting that output-space confidence primarily captures textual plausibility rather than agreement with the image. To address this gap, we propose LookBack, a training-free LVLM response scoring method that augments token likelihood with visual lookback score, a lightweight measure of how strongly each response token refers to image tokens. Across four benchmarks and three models, LookBack consistently improves Best-of-$N$ selection over existing baselines with negligible additional overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。