提出多层级匹配网络,提升多模态实体链接的准确性。
Multi-level Matching Network for Multimodal Entity Linking
- 引入跨模态双向匹配与模内对比学习。
- 在三个数据集上超越现有最佳方法,准确率提升显著。
- 适合研究多模态对齐与知识图谱构建的学者。
多模态实体链接(MEL)旨在将多模态语境中的模糊提及链接到多模态知识库中的对应实体。现有方法多基于表示学习或视觉-语言预训练机制,但存在两大局限:一是忽略同一模态内的负样本;二是缺乏双向跨模态交互机制。为此,本文提出多层级匹配网络(M3EL)。M3EL包含三个模块:(i)多模态特征提取模块,利用多模态编码器提取模态特定表示,并引入模内对比学习子模块,基于单模态差异获得更具区分性的嵌入;(ii)模内匹配网络模块,包含粗粒度全局对全局和细粒度全局对局部两个层次,实现局部与全局的模内交互;(iii)跨模态匹配网络模块,采用文本到视觉与视觉到文本的双向策略,实现双向跨模态交互。在WikiMEL、RichpediaMEL和WikiDiverse数据集上的大量实验表明,M3EL在性能上显著优于当前最先进方法。
原文摘要 · Abstract (English)
Multimodal entity linking (MEL) aims to link ambiguous mentions within multimodal contexts to corresponding entities in a multimodal knowledge base. Most existing approaches to MEL are based on representation learning or vision-and-language pre-training mechanisms for exploring the complementary effect among multiple modalities. However, these methods suffer from two limitations. On the one hand, they overlook the possibility of considering negative samples from the same modality. On the other hand, they lack mechanisms to capture bidirectional cross-modal interaction. To address these issues, we propose a Multi-level Matching network for Multimodal Entity Linking (M3EL). Specifically, M3EL is composed of three different modules: (i) a Multimodal Feature Extraction module, which extracts modality-specific representations with a multimodal encoder and introduces an intra-modal contrastive learning sub-module to obtain better discriminative embeddings based on uni-modal differences; (ii) an Intra-modal Matching Network module, which contains two levels of matching granularity: Coarse-grained Global-to-Global and Fine-grained Global-to-Local, to achieve local and global level intra-modal interaction; (iii) a Cross-modal Matching Network module, which applies bidirectional strategies, Textual-to-Visual and Visual-to-Textual matching, to implement bidirectional cross-modal interaction. Extensive experiments conducted on WikiMEL, RichpediaMEL, and WikiDiverse datasets demonstrate the outstanding performance of M3EL when compared to the state-of-the-art baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。