让大模型自己找关键证据,无需训练就能提升视觉问答准确率
Look Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question Answering

- 利用模型自身注意力识别关键图文内容,过滤干扰信息
- 在四个数据集上平均提升12.5个准确率点,适配多种大模型
- 无需修改模型或训练,适合快速部署到现有系统
基于知识的视觉问答(KB-VQA)要求多模态大语言模型(MLLMs)结合细粒度视觉线索与检索到的文本证据。然而,检索常引入噪声和部分相关的内容,图像中也存在干扰区域,导致预训练的MLLM忽略真正支持答案的证据。为此,我们提出训练免费的推理时框架Look Twice(LoT),将模型内部注意力转化为显式的多模态证据选择机制。LoT首先利用模型内部注意力模式识别与查询相关的图像区域和文本句子,过滤注意力陷阱和干扰内容,并在生成答案前重构输入以明确突出所选证据。该方法无需参数更新、辅助模型或架构修改。在四个KB-VQA基准和十种不同规模(2B至38B参数)的现成MLLM上,LoT提升了所有评估的骨干模型,平均准确率提升最高达+12.5点。结合已有上下文精炼策略后仍能进一步增益,证明其作为通用有效机制,可帮助预训练MLLM更精准利用多模态证据。源码已公开于https://aimagelab.github.io/LoT/。
原文摘要 · Abstract (English)
Knowledge-based Visual Question Answering (KB-VQA) requires Multimodal Large Language Models (MLLMs) to identify and combine fine-grained visual cues with retrieved textual evidence. However, retrieval often introduces noisy and partially relevant content, while images contain distracting visual regions, causing pretrained MLLMs to overlook the evidence that actually supports the answer. To address this, we introduce Look Twice (LoT), a training-free inference-time framework that turns the model's own internal attention into an explicit multimodal evidence-selection mechanism. LoT first leverages the model's internal attention patterns to identify query-relevant image regions and textual sentences, filters attention sinks and distracting content, and reformulates the input to explicitly highlight the selected evidence before answer generation. The method requires no parameter updates, auxiliary models, or architectural modifications. Across four KB-VQA benchmarks and ten off-the-shelf MLLMs ranging from 2B to 38B parameters, LoT improves every evaluated backbone, with average gains of up to +12.5 accuracy points. It also provides further gains when combined with established context-refinement strategies, yielding additional improvements over already refined inputs. These results establish LoT as a general and effective mechanism for enabling pretrained MLLMs to exploit available multimodal evidence more accurately. Source code is publicly available at https://aimagelab.github.io/LoT/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。