提出新框架与评测集,解决多模态知识编辑后推理不连贯问题
Hybrid-DMKG: A Hybrid Reasoning Framework over Dynamic Multimodal Knowledge Graphs for Multimodal Multihop QA with Knowledge Editing
- 用大模型拆解多跳问题,结合图文检索定位更新事实
- 双路径推理:关系链接预测+视觉语言模型检索增强
- 首个支持多模态多跳问答与知识编辑的评测基准MMQAKE
多模态知识编辑(MKE)将传统知识编辑拓展至文本与图像并存的场景。然而现有基准主要评估最终答案正确性,忽视中间推理质量与对视觉改写输入的鲁棒性。为此,我们提出首个支持多模态多跳问答与知识编辑的评测基准MMQAKE,评估模型在2-5跳事实链上的逐步推理能力及对视觉改写输入的鲁棒性。实验表明,当前MKE方法在知识更新后难以持续维护多模态推理链。为此,我们提出Hybrid-DMKG框架,基于动态多模态知识图谱(DMKG),通过大模型分解多跳问题,联合编码子问题与候选实体及其图像进行多模态检索。答案推理采用双路径机制:关系链接预测与基于大视觉语言模型的RAG推理。决策模块融合两路证据生成可信答案。在MMQAKE上,Hybrid-DMKG显著优于现有方法,在准确率和知识更新鲁棒性上均有提升。
原文摘要 · Abstract (English)
Multimodal Knowledge Editing (MKE) extends traditional knowledge editing to settings involving both textual and visual modalities. However, existing MKE benchmarks primarily assess final answer correctness while neglecting the quality of intermediate reasoning and robustness to visually rephrased inputs. To address this limitation, we introduce MMQAKE, the first benchmark for multimodal multihop question answering with knowledge editing. MMQAKE evaluates (1) a model's ability to reason over 2-5-hop factual chains that span both text and images, including performance at each intermediate step, and (2) robustness to visually rephrased inputs in multihop questions. Our evaluation shows that current MKE methods often struggle to consistently update and reason over multimodal reasoning chains after knowledge edits. To overcome these challenges, we propose Hybrid-DMKG, a hybrid reasoning framework built on a dynamic multimodal knowledge graph (DMKG) to enable accurate multihop reasoning over updated multimodal knowledge. Hybrid-DMKG first uses a large language model to decompose multimodal multihop questions into sequential sub-questions, then applies a multimodal retrieval model to locate updated facts by jointly encoding each sub-question with candidate entities and their associated images. For answer inference, a hybrid reasoning module operates over the DMKG via two parallel paths: (1) relation linking prediction, and (2) RAG reasoning with large vision-language models. A decision module aggregates evidence from both paths to select the most credible answer. Experimental results on MMQAKE show that Hybrid-DMKG significantly outperforms existing MKE approaches, achieving higher accuracy and improved robustness to knowledge updates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。