用文本引导的注意力图增强视觉语言模型的图像理解能力
Attention Prompting on Image for Large Vision-Language Models
- 将文本查询生成的注意力热图叠加到原图上作为输入
- 在MM-Vet和LLaVA-Wild上分别提升3.8%和2.9%
- 无需修改模型即可提升多任务表现，适合部署优化
与大型语言模型（LLMs）相比，大型视觉语言模型（LVLMs）可接受图像输入，展现出更丰富的涌现能力，并在多种视觉语言任务中表现优异。受LLMs中文本提示的启发，视觉提示被用于增强LVLM对视觉信息的感知能力。然而，以往的视觉提示技术仅处理视觉输入而忽略文本查询，限制了模型遵循文本指令完成任务的能力。为此，本文提出一种新的提示技术——图像注意力提示（Attention Prompting on Image），仅需将基于文本查询生成的注意力热图叠加到原始图像上，即可有效提升LVLM在各类任务中的表现。具体而言，通过CLIP等辅助模型根据文本查询生成图像注意力热图，再将其与原图像素值相乘得到最终输入。大量实验验证了该方法的有效性：例如，在MM-Vet和LLaVA-Wild基准上，该方法使LLaVA-1.5的性能分别提升3.8%和2.9%。
原文摘要 · Abstract (English)
Compared with Large Language Models (LLMs), Large Vision-Language Models (LVLMs) can also accept images as input, thus showcasing more interesting emergent capabilities and demonstrating impressive performance on various vision-language tasks. Motivated by text prompting in LLMs, visual prompting has been explored to enhance LVLMs' capabilities of perceiving visual information. However, previous visual prompting techniques solely process visual inputs without considering text queries, limiting the models' ability to follow text instructions to complete tasks. To fill this gap, in this work, we propose a new prompting technique named Attention Prompting on Image, which just simply overlays a text-query-guided attention heatmap on the original input image and effectively enhances LVLM on various tasks. Specifically, we generate an attention heatmap for the input image dependent on the text query with an auxiliary model like CLIP. Then the heatmap simply multiplies the pixel values of the original image to obtain the actual input image for the LVLM. Extensive experiments on various vison-language benchmarks verify the effectiveness of our technique. For example, Attention Prompting on Image improves LLaVA-1.5 by 3.8% and 2.9% on MM-Vet and LLaVA-Wild benchmarks, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。