动态调整检索文档数量，提升多模态问答准确率
Self-adaptive Multimodal Retrieval-Augmented Generation
- 根据输入查询自适应筛选图文相关文档
- 在多个数据集上超越现有最佳方法
- 适合需要高精度多模态信息的场景
传统检索增强生成（RAG）方法依赖固定数量的检索文档，常导致信息不全或噪声干扰，影响任务表现。尽管近期自适应方法有所改进，但在复杂真实多模态任务中应用仍有限。为此，我们提出自适应多模态检索增强生成（SAM-RAG），专为多模态场景设计。SAM-RAG能根据输入查询动态过滤相关文档（必要时包含图像描述），并验证检索内容与生成结果的质量。大量实验表明，SAM-RAG在检索准确率和生成质量上均优于现有先进方法。通过消融实验与有效性分析，其在保持高召回质量的同时提升了整体任务性能。代码已开源：https://github.com/SAM-RAG/SAM_RAG。
原文摘要 · Abstract (English)
Traditional Retrieval-Augmented Generation (RAG) methods are limited by their reliance on a fixed number of retrieved documents, often resulting in incomplete or noisy information that undermines task performance. Although recent adaptive approaches alleviated these problems, their application in intricate and real-world multimodal tasks remains limited. To address these, we propose a new approach called Self-adaptive Multimodal Retrieval-Augmented Generation (SAM-RAG), tailored specifically for multimodal contexts. SAM-RAG not only dynamically filters relevant documents based on the input query, including image captions when needed, but also verifies the quality of both the retrieved documents and the output. Extensive experimental results show that SAM-RAG surpasses existing state-of-the-art methods in both retrieval accuracy and response generation. By further ablation experiments and effectiveness analysis, SAM-RAG maintains high recall quality while improving overall task performance in multimodal RAG task. Our codes are available at https://github.com/SAM-RAG/SAM_RAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。