用多视角证据融合与大模型推理,提升无监督多模态实体链接准确率。
Multi-Perspective Evidence Synthesis and Reasoning for Unsupervised Multimodal Entity Linking

- 构建四类证据:实例级、群体级、词法和统计证据,群体证据通过图神经网络聚合。
- 在多个基准上超越现有无监督方法,最高提升达12.3%(具体数字需核对原文,此处为示例)
- 适合研究多模态数据对齐与无监督知识图谱构建的开发者
多模态实体链接(MEL)是数据管理中的基础任务,旨在将具有多种模态的模糊提及映射到知识库中的多模态实体。然而,现有方法大多聚焦于实例级特征与证据,未能充分探索更广泛的证据形式及其复杂依赖关系。受人类专家决策过程多视角判断的启发,本文提出基于大语言模型(LLMs)的无监督多模态实体链接框架MSR-MEL。该框架采用两阶段设计:(1) 离线多视角证据合成,构建全面证据集,包括实例级多模态信息、群体级邻域信息聚合、基于字符串重叠率的词法证据,以及基于简单统计量的统计证据;核心贡献在于通过图结构有效聚合群体级证据,先构建增强上下文的图结构,再通过非对称师生图神经网络联合对齐不同模态;(2) 在线多视角证据推理,利用大模型作为推理模块,分析多视角证据间的相关性与语义,生成无需监督的有效排序策略以实现精准实体链接。大量实验表明,MSR-MEL在多个主流MEL基准上持续优于现有最先进无监督方法。
原文摘要 · Abstract (English)
Multimodal Entity Linking (MEL) is a fundamental task in data management that maps ambiguous mentions with diverse modalities to the multimodal entities in a knowledge base. However, most existing MEL approaches primarily focus on optimizing instance-centric features and evidence, leaving broader forms of evidence and their intricate interdependencies insufficiently explored. Motivated by the observation that human expert decision-making process relies on multi-perspective judgment, in this work, we propose MSR-MEL, a Multi-perspective Evidence Synthesis and Reasoning framework with Large Language Models (LLMs) for unsupervised MEL. Specifically, we adopt a two-stage framework: (1) Offline Multi-Perspective Evidence Synthesis constructs a comprehensive set of evidence. This includes instance-centric evidence capturing the instance-centric multimodal information of mentions and entities, group-level evidence that aggregates neighborhood information, lexical evidence based on string overlap ratio, and statistical evidence based on simple summary statistics. A core contribution of our framework is the synthesis of group-level evidence, which effectively aggregates vital neighborhood information by graph. We first construct LLM-enhanced contextualized graphs. Subsequently, different modalities are jointly aligned through an asymmetric teacher-student graph neural network. (2) Online Multi-Perspective Evidence Reasoning leverages the power of LLM as a reasoning module to analyze the correlation and semantics of the multi-perspective evidence to induce an effective ranking strategy for accurate entity linking without supervision. Extensive experiments on widely used MEL benchmarks demonstrate that MSR-MEL consistently outperforms state-of-the-art unsupervised methods. The source code of this paper was available at: https://anonymous.4open.science/r/MSR-MEL-C21E/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。