用重排摘要后的图像描述提升大模型的视觉问答能力
SCRA-VQA: Summarized Caption-Rerank for Augmented Large Language Models in Visual Question Answering
- 用视觉语言模型生成并精简图像描述，去除无关噪声
- 在两个数据集上分别达到38.8%和34.6%准确率
- 无需昂贵训练，适合想提升现有大模型推理能力的研究者
获取高质量知识是基于知识的视觉问答（KB-VQA）的核心挑战。近期方法使用大语言模型（LLM）作为知识引擎，通常依赖图像描述辅助理解图像。但这些描述常包含与问题无关的噪声，且大模型缺乏对VQA任务的理解，限制了推理能力。为此，我们提出SCRA-VQA：利用预训练视觉语言模型生成图像描述，并同步生成上下文示例，对描述进行摘要与重排，剔除无关信息。该重排机制使大模型更精准理解图像与问题，显著增强推理能力与任务适应性，且无需昂贵的端到端训练。基于6.7B参数的大模型，SCRA-VQA在两个挑战性数据集OK-VQA和A-OKVQA上分别取得38.8%和34.6%的准确率。代码已开源。
原文摘要 · Abstract (English)
Acquiring high-quality knowledge is a central focus in Knowledge-Based Visual Question Answering (KB-VQA). Recent methods use large language models (LLMs) as knowledge engines for answering. These methods generally employ image captions as visual text descriptions to assist LLMs in interpreting images. However, the captions frequently include excessive noise irrelevant to the question, and LLMs generally do not comprehend VQA tasks, limiting their reasoning capabilities. To address this issue, we propose the Summarized Caption-Rerank Augmented VQA (SCRA-VQA), which employs a pre-trained visual language model to convert images into captions. Moreover, SCRA-VQA generates contextual examples for the captions while simultaneously summarizing and reordering them to exclude unrelated information. The caption-rerank process enables LLMs to understand the image information and questions better, thus enhancing the model's reasoning ability and task adaptability without expensive end-to-end training. Based on an LLM with 6.7B parameters, SCRA-VQA performs excellently on two challenging knowledge-based VQA datasets: OK-VQA and A-OKVQA, achieving accuracies of 38.8% and 34.6%. Our code is available at https://github.com/HubuKG/SCRA-VQA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。