让多模态大模型识别图文冲突并自适应生成更可靠的回应。
CAER: Conflict-Aware Evidence Routing with Dual Prefix Experts for Multimodal Large Language Models
- 用软查询从视觉特征中检索证据，精准定位图文矛盾点。
- 双前缀专家路由机制，分别处理支持与矛盾输入，提升生成可靠性。
- 无需修改主干模型，在公开数据集和农业冲突数据集上均有效。
多模态大语言模型在多模态理解与生成方面表现卓越，但在文本与视觉证据冲突时仍易产生幻觉，输出与视觉内容不符。现有方法多依赖解码策略、额外训练、验证或提示技术，但缺乏细粒度的冲突定位与感知生成能力。本文提出CAER，一种不依赖主干模型的视觉-语言冲突检测与感知生成框架。CAER引入基于片段的证据路由器，将陈述表示转换为软文本查询，从冻结的视觉标记中检索对应证据，实现细粒度冲突估计。此外，设计双前缀专家路由机制，分别学习支持视觉证据和与之矛盾输入的专家，通过显式专家选择实现冲突感知生成。在公开的MMMC基准及新构建的AgriConflict数据集上的实验表明，CAER能有效检测视觉-语言冲突，并在不更新主干参数的情况下提升开源多模态大模型的可靠性。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in multimodal understanding and generation. However, when textual inputs conflict with visual evidence, they still suffer from hallucinations and produce responses inconsistent with visual content. Existing approaches mainly rely on decoding strategies, additional training, verification methods, or prompting techniques, but often lack fine-grained conflict localization and conflict-aware generation. In this work, we propose CAER, a backbone-agnostic framework for visual-language conflict detection and conflict-aware generation. CAER introduces a span-grounded evidence router that transforms claim representations into soft textual queries and retrieves corresponding evidence from frozen visual tokens, enabling fine-grained conflict estimation. Furthermore, we design a dual-prefix expert routing mechanism that learns separate experts for visually supported and contradicted inputs, enabling conflict-aware generation through explicit expert selection. Experiments on the public MMMC benchmark and our newly curated AgriConflict dataset demonstrate that CAER effectively detects visual-language conflicts and improves the reliability of open-source MLLMs without updating their backbone parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。