用知识图谱增强图文实体链接，提升准确率。
KGMEL: Knowledge Graph-Enhanced Multimodal Entity Linking
- 利用视觉语言模型生成提及的高质量知识图谱三元组。
- 通过对比学习融合文本、图像和三元组信息，精准召回候选实体。
- 结合大模型重排三元组，实现更优的实体匹配，适合多模态研究者。
实体链接（EL）将文本提及与知识库中的对应实体对齐，广泛应用于语义搜索和问答系统。近年来，多模态实体链接（MEL）通过结合文本与图像，有效降低歧义并提升对齐精度。然而，现有方法大多忽视了知识图谱（KG）中丰富的结构化三元组信息。本文提出KGMEL框架，分三阶段利用KG三元组增强MEL：(1) 生成阶段，基于文本与图像，使用视觉语言模型为每个提及生成高质量三元组；(2) 检索阶段，通过对比学习构建融合文本、图像及三元组的联合提及-实体表示，检索候选实体；(3) 重排序阶段，优化候选实体的三元组，并借助大语言模型确定最佳匹配实体。在多个基准数据集上的实验表明，KGMEL显著优于现有方法。代码与数据集已开源：https://github.com/juyeonnn/KGMEL。
原文摘要 · Abstract (English)
Entity linking (EL) aligns textual mentions with their corresponding entities in a knowledge base, facilitating various applications such as semantic search and question answering. Recent advances in multimodal entity linking (MEL) have shown that combining text and images can reduce ambiguity and improve alignment accuracy. However, most existing MEL methods overlook the rich structural information available in the form of knowledge-graph (KG) triples. In this paper, we propose KGMEL, a novel framework that leverages KG triples to enhance MEL. Specifically, it operates in three stages: (1) Generation: Produces high-quality triples for each mention by employing vision-language models based on its text and images. (2) Retrieval: Learns joint mention-entity representations, via contrastive learning, that integrate text, images, and (generated or KG) triples to retrieve candidate entities for each mention. (3) Reranking: Refines the KG triples of the candidate entities and employs large language models to identify the best-matching entity for the mention. Extensive experiments on benchmark datasets demonstrate that KGMEL outperforms existing methods. Our code and datasets are available at: https://github.com/juyeonnn/KGMEL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。