arXiv:2607.15592cs.AI2026-07

用大模型引导扩散,让多模态知识图谱补全更准

MGDT: MLLM-Guided Diffusion Transformer with Relation-Adaptive Mixture-of-Experts for Multimodal Knowledge Graph Completion

论文配图:MGDT: MLLM-Guided Diffusion Transformer with Relation-Adaptive Mixture-of-Experts for Multimodal Knowledge Graph Completion
图 1 · 摘自论文原文
  • 先选相关模态路径,再用大模型对齐语义
  • 在三个数据集上超越现有方法,效果显著提升
  • 适合做多模态知识图谱补全的科研与工程人员

多模态知识图谱补全(MKGC)需从结构、文本和视觉线索中推断缺失实体。现有基于扩散的方法直接在原始多模态特征上去噪,迫使去噪器同时完成关系相关线索选择、跨模态语义对齐和结构感知实体生成,导致扩散过程存在噪声与语义不一致问题,影响补全性能。为此,我们提出MGDT:基于关系自适应混合专家(RASR-MoE)的多模态大模型引导扩散变换器,采用先对齐后扩散范式。MGDT首先通过RASR-MoE模块选择与关系相关的多模态语义转换路径,并抑制无关模态干扰;接着利用冻结的多模态大语言模型(MLLM)作为语义锚点,将路由后的多模态表示对齐至统一潜在空间,降低跨模态语义异质性;最后,知识图谱扩散变换器(KGDT)在对齐空间中进行图条件去噪生成,输出缺失实体表示。在三个基准数据集上的实验表明,MGDT持续优于强基线。

原文摘要 · Abstract (English)

Multimodal Knowledge Graph Completion (MKGC) requires inferring missing entities from structural, textual, and visual cues. Existing diffusion-based MKGC methods usually denoise directly on raw multimodal features. Such a design forces the denoiser to simultaneously perform relation-dependent cue selection, cross-modal semantic alignment, and structure-aware entity generation, which introduces noisy and semantically inconsistent conditions for diffusion and consequently leads to suboptimal completion performance. To address this limitation, we propose MGDT: MLLM-Guided Diffusion Transformer with Relation-Adaptive Mixture-of-Experts (MGDT), a novel MKGC framework built on an align-then-diffuse paradigm. MGDT first employs a Relation-Adaptive Semantic Routing Mixture-of-Experts (RASR-MoE) module to select relation-relevant multimodal semantic transformation paths and suppress irrelevant modality interference. MGDT then uses a frozen Multimodal Large Language Model (MLLM) as a semantic anchor to align the routed multimodal representations into a unified latent space and reduce cross-modal semantic heterogeneity. Finally, a Knowledge Graph Diffusion Transformer (KGDT) performs graph-conditioned denoising generation in the aligned space to produce the missing entity representation. Experiments on three benchmark datasets show that MGDT consistently outperforms strong baselines.

知识图谱多模态扩散模型大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。