用强化学习生成难负样本,提升多模态实体链接效果
PGMEL: Policy Gradient-based Generative Adversarial Network for Multimodal Entity Linking
- 用策略梯度优化生成器,自动构造难负样本
- 在三个数据集上超越当前最优方法
- 适合需要高质量负样本的多模态匹配任务
实体链接任务旨在将文本中的提及与知识图谱中的实体关联。近年来,多模态实体链接(MEL)技术通过融合文本与视觉信息学习更全面的嵌入表示而受到关注。负样本质量对度量学习至关重要,但现有研究尚未探索此方向。为此,本文提出基于策略梯度的生成对抗网络(PGMEL),在生成对抗框架下,由生成器负责生成高质量负样本,判别器执行度量学习。由于生成过程为离散操作,采用策略梯度进行优化。在Wiki-MEL、Richpedia-MEL和WikiDiverse数据集上的实验表明,PGMEL通过选择具有挑战性的负样本,学习到更具语义意义的表示,性能优于现有先进方法。
原文摘要 · Abstract (English)
The task of entity linking, which involves associating mentions with their respective entities in a knowledge graph, has received significant attention due to its numerous potential applications. Recently, various multimodal entity linking (MEL) techniques have been proposed, targeted to learn comprehensive embeddings by leveraging both text and vision modalities. The selection of high-quality negative samples can potentially play a crucial role in metric/representation learning. However, to the best of our knowledge, this possibility remains unexplored in existing literature within the framework of MEL. To fill this gap, we address the multimodal entity linking problem in a generative adversarial setting where the generator is responsible for generating high-quality negative samples, and the discriminator is assigned the responsibility for the metric learning tasks. Since the generator is involved in generating samples, which is a discrete process, we optimize it using policy gradient techniques and propose a policy gradient-based generative adversarial network for multimodal entity linking (PGMEL). Experimental results based on Wiki-MEL, Richpedia-MEL and WikiDiverse datasets demonstrate that PGMEL learns meaningful representation by selecting challenging negative samples and outperforms state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。