让多模态大模型按需检索，自动选模态，生成更准更快。
Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generation
- 根据问题动态决定是否检索、选什么模态
- 生成质量提升17.07%，检索时间减少8.95%
- 适合需要实时知识的多模态问答场景
多模态检索增强生成（MRAG）通过引入外部知识库的非参数化知识，提升了多模态大语言模型（MLLM）生成内容的事实性和时效性。然而现有方法存在检索策略静态、模态选择僵化、信息利用不充分等问题，导致三个关键挑战：何时检索、用何种模态、如何有效使用。为此，我们提出Windsock，一个查询相关的模块，用于判断检索必要性与模态选择，显著降低计算开销并提升响应质量。同时，我们设计了动态抗噪指令微调（DANCE），增强模型对检索信息的利用能力并保持对噪声的鲁棒性。此外，我们采用基于MLLM内部知识的自评估方法，将问答数据集转换为MRAG训练数据。大量实验表明，该方法使生成质量提升17.07%，同时减少8.95%的检索时间。
原文摘要 · Abstract (English)
Multimodal Retrieval-Augmented Generation (MRAG) has emerged as a promising method to generate factual and up-to-date responses of Multimodal Large Language Models (MLLMs) by incorporating non-parametric knowledge from external knowledge bases. However, existing MRAG approaches suffer from static retrieval strategies, inflexible modality selection, and suboptimal utilization of retrieved information, leading to three critical challenges: determining when to retrieve, what modality to incorporate, and how to utilize retrieved information effectively. To address these challenges, we introduce Windsock, a query-dependent module making decisions on retrieval necessity and modality selection, effectively reducing computational overhead and improving response quality. Additionally, we propose Dynamic Noise-Resistance (DANCE) Instruction Tuning, an adaptive training strategy that enhances MLLMs' ability to utilize retrieved information while maintaining robustness against noise. Moreover, we adopt a self-assessment approach leveraging knowledge within MLLMs to convert question-answering datasets to MRAG training datasets. Extensive experiments demonstrate that our proposed method significantly improves the generation quality by 17.07% while reducing 8.95% retrieval times.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。