用四阶段框架让大模型更准答图问，无需训练
GC-KBVQA: A New Four-Stage Framework for Enhancing Knowledge Based Visual Question Answering Performance
- 先生成紧扣问题的图文描述，再融合外部知识构造精准提示
- 零样本下在多个数据集上超越现有方法，无需微调
- 适合想快速部署通用视觉问答系统的研究者和开发者
基于知识的视觉问答（KB-VQA）需依赖图像之外的信息进行推理。早期方法使用显式知识库，近期则利用大语言模型（LLM）作为隐式知识源。但辅助文本常与问题无关或含干扰信息，限制了性能。本文提出四阶段框架GC-KBVQA，通过生成问题相关的图文描述，结合外部知识构建高信息量提示，使预训练LLM在零样本条件下完成多种视觉问答任务，无需端到端多模态训练。该方法不依赖特定任务微调，显著降低部署成本。实验表明其性能显著优于现有方法，代码将公开。
原文摘要 · Abstract (English)
Knowledge-Based Visual Question Answering (KB-VQA) methods focus on tasks that demand reasoning with information extending beyond the explicit content depicted in the image. Early methods relied on explicit knowledge bases to provide this auxiliary information. Recent approaches leverage Large Language Models (LLMs) as implicit knowledge sources. While KB-VQA methods have demonstrated promising results, their potential remains constrained as the auxiliary text provided may not be relevant to the question context, and may also include irrelevant information that could misguide the answer predictor. We introduce a novel four-stage framework called Grounding Caption-Guided Knowledge-Based Visual Question Answering (GC-KBVQA), which enables LLMs to effectively perform zero-shot VQA tasks without the need for end-to-end multimodal training. Innovations include grounding question-aware caption generation to move beyond generic descriptions and have compact, yet detailed and context-rich information. This is combined with knowledge from external sources to create highly informative prompts for the LLM. GC-KBVQA can address a variety of VQA tasks, and does not require task-specific fine-tuning, thus reducing both costs and deployment complexity by leveraging general-purpose, pre-trained LLMs. Comparison with competing KB-VQA methods shows significantly improved performance. Our code will be made public.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。