用视觉语言模型生成跨模态上下文,让Transformer更高效完成多模态知识图谱补全。
Transformer-Based Multimodal Knowledge Graph Completion with Link-Aware Contexts
- 用预训练视觉语言模型将实体图像转为文本序列,构建跨模态上下文。
- 在多个大规模数据集上达到竞争性性能,模型规模显著小于传统方法。
- 无需复杂调参,适合需要轻量高效多模态推理的场景。
多模态知识图谱补全(MMKGC)旨在利用多种模态信息与结构数据预测知识图谱中的缺失链接。现有方法多基于传统知识图谱嵌入(KGE)模型,需为每个实体创建嵌入,导致模型庞大且难以高效融合多模态信息,尤其在真实世界图谱中表现不佳。尽管Transformer在单模态知识图谱补全(KGC)中表现优异,但其跨模态能力有限。近期大型视觉-语言模型(VLMs)在跨模态任务中展现潜力,但训练成本高昂。本文提出一种新方法,将基于Transformer的KGE模型与预训练VLM生成的跨模态上下文结合,扩展其在MMKGC中的应用。具体地,我们利用预训练VLM将实体及其邻域的相关视觉信息转化为文本序列,并将KGC建模为序列到序列任务,通过生成的跨模态上下文对模型进行微调。该方法简单有效,在多个大规模数据集上实现竞争性性能,同时显著降低模型规模,仅需少量超参数调整。
原文摘要 · Abstract (English)
Multimodal knowledge graph completion (MMKGC) aims to predict missing links in multimodal knowledge graphs (MMKGs) by leveraging information from various modalities alongside structural data. Existing MMKGC approaches primarily extend traditional knowledge graph embedding (KGE) models, which often require creating an embedding for every entity. This results in large model sizes and inefficiencies in integrating multimodal information, particularly for real-world graphs. Meanwhile, Transformer-based models have demonstrated competitive performance in knowledge graph completion (KGC). However, their focus on single-modal knowledge limits their capacity to utilize cross-modal information. Recently, Large vision-language models (VLMs) have shown potential in cross-modal tasks but are constrained by the high cost of training. In this work, we propose a novel approach that integrates Transformer-based KGE models with cross-modal context generated by pre-trained VLMs, thereby extending their applicability to MMKGC. Specifically, we employ a pre-trained VLM to transform relevant visual information from entities and their neighbors into textual sequences. We then frame KGC as a sequence-to-sequence task, fine-tuning the model with the generated cross-modal context. This simple yet effective method significantly reduces model size compared to traditional KGE approaches while achieving competitive performance across multiple large-scale datasets with minimal hyperparameter tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。