利用大模型内部表示精准定位图像细节，提升细粒度视觉问答效率。
FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering
- 基于大模型内部键值缓存生成目标相关性图，指导高效搜索
- 在4个数据集上超越现有方法，准确率更高且计算量减少3至6.5倍
- 无需微调，兼容主流注意力机制，适合快速部署到各类多模态模型
尽管多模态大语言模型（MLLMs）具备强大的图像-文本感知与推理能力，针对图像细微内容的视觉问答（VQA）仍具挑战。现有视觉裁剪方法存在需任务特定微调、因盲目遍历导致效率低、或不兼容高效注意力实现等问题。本文提出无需训练的视觉裁剪方法FOCUS，利用MLLM内部表示引导搜索最相关图像区域。流程包括：第一步识别提示中的目标对象；第二步使用键值缓存（KV cache）计算对象相关性图；第三步基于该图排序并筛选相关图像区域；第四步仅用排名靠前区域完成细粒度VQA任务。得益于这一有向搜索策略，FOCUS在四个细粒度VQA数据集及三类MLLM上表现优异，显著优于三种主流裁剪方法，在精度和效率上均领先，同时达到最佳基线模型ZoomEye的水平，但所需计算量减少3至6.5倍。
原文摘要 · Abstract (English)
While Multimodal Large Language Models (MLLMs) offer strong perception and reasoning capabilities for image-text input, Visual Question Answering (VQA) focusing on small image details still remains a challenge. Although visual cropping techniques seem promising, recent approaches have several limitations: the need for task-specific fine-tuning, low efficiency due to uninformed exhaustive search, or incompatibility with efficient attention implementations. We address these shortcomings by proposing a training-free visual cropping method, dubbed FOCUS, that leverages MLLM-internal representations to guide the search for the most relevant image region. This is accomplished in four steps: first, we identify the target object(s) in the VQA prompt; second, we compute an object relevance map using the key-value (KV) cache; third, we propose and rank relevant image regions based on the map; and finally, we perform the fine-grained VQA task using the top-ranked region. As a result of this informed search strategy, FOCUS achieves strong performance across four fine-grained VQA datasets and three types of MLLMs. It outperforms three popular visual cropping methods in both accuracy and efficiency, and matches the best-performing baseline, ZoomEye, while requiring 3 - 6.5 x less compute.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。