提出新方法解决多模态检索中不可靠信息干扰问题,提升问答准确性。
Trust Before Fusion: QIMG-7 and Source-Aware Resolution for Polluted Multimodal RAG

- 设计QIMG-7基准测试,覆盖7类图像攻击和16种污染场景
- 污染后全模态融合准确率从0.908降至0.490,退化严重
- 提出源感知可信度判断机制,不需训练即可提升效果
多模态检索增强生成(RAG)常在干净证据下评估,但真实检索可能返回主题相关却不可靠的内容:如虚假文本、误导性图像,来自损坏元数据、实体替换、拼写叠加、语义编辑、对抗补丁、混合或风格迁移等。本文提出QIMG-7,一个面向多句事实问答的受控多模态检索污染基准,涵盖四个数据集、七类图像攻击及16组清洁/污染对比设置,每种方法共1,760个评测样本。在四个生成器/门控组合中,直接多模态融合表现脆弱:以gpt-4o-mini为主模型时,全模态支持率从清洁文本下的0.908降至污染后0.490,参数化回退反而更安全。为此提出无需训练的源感知可信度分辨(SATR),通过比较参数化、仅文本与全模态候选答案,依据来源可靠性选择或回退。其中场选择器变体取得最佳平衡得分0.816,较全模态提升11.7分,优于级联路由器2.7分。消融实验表明,在以文本优先的设定中,显式建模文本可信度是主要增益来源。总体而言,在文本优先的事实问答中存在多模态检索冲突时,应采取选择性信任而非无条件融合。代码与数据见https://github.com/SaadElDine/Trust_Before_Fusion。
原文摘要 · Abstract (English)
Multimodal retrieval-augmented generation (RAG) is often evaluated with clean evidence, yet real retrieval can return topically relevant but unreliable content: false text and misleading images from corrupted metadata, entity swaps, typographic overlays, semantic edits, adversarial patches, blends, or style transfer. We introduce QIMG-7, a controlled benchmark for multimodal retrieval pollution in multi-sentence factual QA, spanning four datasets, seven image-attack families, and 16 paired clean/polluted regimes, for 1,760 evaluation rows per method. Across four generator/gate stacks, naive multimodal fusion is brittle: in the main gpt-4o-mini stack, Full-MM support drops from 0.908 with clean text to 0.490 with polluted text, often making Parametric fallback safer than retrieval. We propose source-aware trust resolution (SATR), a training-free approach that compares Parametric, Text-only, and Full-MM candidate answers and selects among candidate answers or falls back based on source reliability. The Field-Selector variant achieves the best balanced score, 0.816, improving over Full-MM by 11.7 points and over the Cascaded Router by 2.7 points. Ablations show that, in this text-first setting, explicit text-reliability modeling is the dominant driver of these gains. Overall, in text-first factual QA with multimodal retrieval conflict, our results support selective trust rather than unconditional fusion. Artifacts are available at https://github.com/SaadElDine/Trust_Before_Fusion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。