解决多模态问答中知识来源不一致的问题，提升答案可靠性。
CoRe-MMRAG: Cross-Source Knowledge Reconciliation for Multimodal RAG
- 通过四阶段流程融合模型内知识与外部多模态证据
- 在InfoSeek和Encyclopedic-VQA上分别提升5.6%和9.3%
- 适合需要高可信多模态推理的应用场景
多模态检索增强生成（MMRAG）通过引入外部多模态知识来增强多模态大模型，但面临两大挑战：参数化知识与检索知识之间的不一致性（PRKI），导致可靠性判断困难；视觉与文本知识之间的错位（VTKI），破坏实体表征。为此，我们提出跨源知识协调框架CoRe-MMRAG，采用四阶段端到端流程：先基于参数知识生成内部回答，再通过联合相似性评估选择最相关多模态证据，生成外部回答，最后融合二者输出可靠答案。此外，专门的训练范式增强了知识源区分、多模态融合与统一答案生成能力。在KB-VQA基准上的实验表明，CoRe-MMRAG相比基线方法显著提升性能，在InfoSeek和Encyclopedic-VQA上分别取得5.6%和9.3%的增益。
原文摘要 · Abstract (English)
Multimodal Retrieval-Augmented Generation (MMRAG) has been introduced to enhance Multimodal Large Language Models by incorporating externally retrieved multimodal knowledge, but it introduces two challenges: Parametric-Retrieved Knowledge Inconsistency (PRKI), where discrepancies between parametric and retrieved knowledge create uncertainty in determining reliability, and Visual-Textual Knowledge Inconsistency (VTKI), where misalignment between visual and textual sources disrupts entity representation. To address these challenges, we propose Cross-source knowledge \textbf{Re}conciliation for Multimodal RAG (CoRe-MMRAG), a novel end-to-end framework that effectively reconciles inconsistencies across knowledge sources. CoRe-MMRAG follows a four-stage pipeline: it first generates an internal response from parametric knowledge, then selects the most relevant multimodal evidence via joint similarity assessment, generates an external response, and finally integrates both to produce a reliable answer. Additionally, a specialized training paradigm enhances knowledge source discrimination, multimodal integration, and unified answer generation. Experiments on KB-VQA benchmarks show that CoRe-MMRAG achieves substantial improvements over baseline methods, achieving 5.6% and 9.3% performance gains on InfoSeek and Encyclopedic-VQA, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。