用生成式扩散模型补全多模态知识图谱,提升结构感知能力。
DiffusionCom: Structure-Aware Multimodal Diffusion Model for Multimodal Knowledge Graph Completion
- 基于生成式扩散模型,将补全任务建模为从噪声逐步生成联合概率分布的过程。
- 在FB15k-237-IMG和WN18-IMG上超越现有最佳模型,显著提升补全精度。
- 适合作为多模态知识图谱研究者与开发者的技术参考。
当前多数多模态知识图谱补全(MKGC)方法依赖判别式模型,通过最大化条件似然进行学习,难以高效捕捉真实世界知识图谱中的复杂关联,限制了整体性能。为此,本文提出结构感知的多模态扩散模型DiffusionCom,从生成式建模视角出发,将(头实体,关系)对与候选尾实体之间的关联建模为联合概率分布 $p((head, relation), (tail))$,并将MKGC任务视为从噪声逐步生成该分布的过程。为进一步利用知识图谱的结构信息,提出Structure-MKGformer,一种自适应的、结构感知的多模态表示学习方法,作为DiffusionCom的编码器。该方法通过多模态图注意力网络(MGAT)捕获丰富结构信息,并自适应融合至实体表示中,增强其结构感知能力。该设计有效缓解了现有基于多模态预训练模型的MKGC方法在结构信息利用上的不足。DiffusionCom采用生成与判别双损失训练生成器,特征提取器仅使用判别损失优化,兼顾两类模型优势。在FB15k-237-IMG与WN18-IMG数据集上的大量实验表明,DiffusionCom显著优于当前最优模型。
原文摘要 · Abstract (English)
Most current MKGC approaches are predominantly based on discriminative models that maximize conditional likelihood. These approaches struggle to efficiently capture the complex connections in real-world knowledge graphs, thereby limiting their overall performance. To address this issue, we propose a structure-aware multimodal Diffusion model for multimodal knowledge graph Completion (DiffusionCom). DiffusionCom innovatively approaches the problem from the perspective of generative models, modeling the association between the $(head, relation)$ pair and candidate tail entities as their joint probability distribution $p((head, relation), (tail))$, and framing the MKGC task as a process of gradually generating the joint probability distribution from noise. Furthermore, to fully leverage the structural information in MKGs, we propose Structure-MKGformer, an adaptive and structure-aware multimodal knowledge representation learning method, as the encoder for DiffusionCom. Structure-MKGformer captures rich structural information through a multimodal graph attention network (MGAT) and adaptively fuses it with entity representations, thereby enhancing the structural awareness of these representations. This design effectively addresses the limitations of existing MKGC methods, particularly those based on multimodal pre-trained models, in utilizing structural information. DiffusionCom is trained using both generative and discriminative losses for the generator, while the feature extractor is optimized exclusively with discriminative loss. This dual approach allows DiffusionCom to harness the strengths of both generative and discriminative models. Extensive experiments on the FB15k-237-IMG and WN18-IMG datasets demonstrate that DiffusionCom outperforms state-of-the-art models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。