用大模型生成式推理补全多模态知识图谱,效果优于现有方法。
HERGC: Heterogeneous Experts Representation and Generative Completion for Multimodal Knowledge Graphs
- 分步处理:先检索候选三元组,再用大模型生成推理答案
- 在三个基准数据集上超越已有方法,显著提升补全准确率
- 适合研究多模态推理与大模型应用的学者使用
多模态知识图谱(MMKG)通过引入图像、文本等多源信息,丰富了传统知识图谱。多模态知识图谱补全(MMKGC)旨在利用这些异构信号推断缺失事实,以缓解MMKG固有的不完备性。现有方法通常基于封闭世界假设,仅依赖图内信息并采用判别式训练目标,限制了其推理能力。近期的大语言模型(LLMs)凭借大规模参数和海量语料预训练,展现出强大的跨任务推理能力,但在MMKGC中的潜力尚未被充分探索。为此,我们提出HERGC——一种灵活的异构专家表征与生成补全框架。HERGC首先通过异构专家表征检索器融合多模态信息,并为每个不完整三元组生成紧凑候选集;随后利用生成式LLM预测器(支持上下文学习或轻量微调)从候选集中精确识别正确答案。在三个标准MMKG基准上的大量实验表明,HERGC在性能与鲁棒性方面均表现出色,显著优于现有方法。
原文摘要 · Abstract (English)
Multimodal knowledge graphs (MMKGs) enrich traditional knowledge graphs (KGs) by incorporating diverse modalities such as images and text. multimodal knowledge graph completion (MMKGC) seeks to exploit these heterogeneous signals to infer missing facts, thereby mitigating the intrinsic incompleteness of MMKGs. Existing MMKGC methods typically leverage only the information contained in the MMKGs under the closed-world assumption and adopt discriminative training objectives, which limits their reasoning capacity during completion. Recent large language models (LLMs), empowered by massive parameter scales and pretraining on vast corpora, have demonstrated strong reasoning abilities across various tasks. However, their potential in MMKGC remains largely unexplored. To bridge this gap, we propose HERGC, a flexible Heterogeneous Experts Representation and Generative Completion framework for MMKGs. HERGC first deploys a Heterogeneous Experts Representation Retriever that enriches and fuses multimodal information and retrieves a compact candidate set for each incomplete triple. It then uses a Generative LLM Predictor, implemented via either in-context learning or lightweight fine-tuning, to accurately identify the correct answer from these candidates. Extensive experiments on three standard MMKG benchmarks demonstrate HERGC's effectiveness and robustness, achieving superior performance over existing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。