教大模型有选择地用检索信息，提升准确性和抗干扰能力。
SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information
- 通过正负参考样本训练模型，学会筛选有用信息。
- 在7个数据集上显著提升多模态检索利用效果。
- 适合需要可靠外部知识的大模型应用开发者。
大型视觉语言模型（LVLMs）在计算机视觉与自然语言处理交叉领域发挥关键作用。然而，其基于检索的生成能力尚未被充分挖掘：现有方法或仅关注文本模态，或局限于特定任务；多数模型难以有选择地利用检索信息，且易受无关或误导性参考影响。为此，我们提出SURf——一种自精炼框架，旨在教会LVLMs有选择地利用检索信息。当基础模型回答错误时，我们获取能纠正答案的正向参考和无法纠正的负向参考，并结合两者对模型进行微调。在三个任务、七个数据集上的实验表明，该框架显著增强了模型有效利用多模态检索信息的能力，并提升了对无关或误导性信息的鲁棒性。代码已开源：https://github.com/GasolSun36/SURf。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) have become pivotal at the intersection of computer vision and natural language processing. However, the full potential of LVLMs Retrieval-Augmented Generation (RAG) capabilities remains underutilized. Existing works either focus solely on the text modality or are limited to specific tasks. Moreover, most LVLMs struggle to selectively utilize retrieved information and are sensitive to irrelevant or misleading references. To address these challenges, we propose a self-refinement framework designed to teach LVLMs to Selectively Utilize Retrieved Information (SURf). Specifically, when given questions that are incorrectly answered by the LVLM backbone, we obtain references that help correct the answers (positive references) and those that do not (negative references). We then fine-tune the LVLM backbone using a combination of these positive and negative references. Our experiments across three tasks and seven datasets demonstrate that our framework significantly enhances LVLMs ability to effectively utilize retrieved multimodal references and improves their robustness against irrelevant or misleading information. The source code is available at https://github.com/GasolSun36/SURf.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。