统一提取图文关系,支持多种三元组类型。
REMOTE: A Unified Multimodal Relation Extraction Framework with Multilevel Optimal Transport and Mixture-of-Experts
- 用专家混合机制动态选择跨模态交互特征。
- 多层级最优传输保留低层信息,提升表示能力。
- 适用于需要同时处理文本与图像关系的研究者。
多模态关系抽取(MRE)是知识图谱与多媒体领域的重要任务,对构建多模态知识图谱具有关键作用。然而,现有方法通常仅能提取单一类型的三元组,难以扩展至其他关系类型;直接组合这些方法无法捕捉动态跨模态交互,且带来显著计算冗余。为此,我们提出一种统一的多模态关系抽取框架 REMOTE,可同时提取文本实体与视觉对象之间的模内和模间关系。为动态选择不同三元组类型的最佳交互特征,引入专家混合机制,确保最相关模态信息被利用。此外,针对现有编码器多层序列编码常导致低层信息丢失的问题,采用多层级最优传输融合模块,在保持多层编码的同时保留低层特征,生成更具表现力的表征。相应地,我们还构建了统一多模态关系抽取数据集(UMRE),涵盖头尾实体来自文本或图像的各种场景。大量实验表明,REMOTE 能有效提取多种关系三元组,在两个公开 MRE 数据集上几乎所有指标均达到当前最优性能。代码已开源:https://github.com/Nikol-coder/REMOTE。
原文摘要 · Abstract (English)
Multimodal relation extraction (MRE) is a crucial task in the fields of Knowledge Graph and Multimedia, playing a pivotal role in multimodal knowledge graph construction. However, existing methods are typically limited to extracting a single type of relational triplet, which restricts their ability to extract triplets beyond the specified types. Directly combining these methods fails to capture dynamic cross-modal interactions and introduces significant computational redundancy. Therefore, we propose a novel \textit{unified multimodal Relation Extraction framework with Multilevel Optimal Transport and mixture-of-Experts}, termed REMOTE, which can simultaneously extract intra-modal and inter-modal relations between textual entities and visual objects. To dynamically select optimal interaction features for different types of relational triplets, we introduce mixture-of-experts mechanism, ensuring the most relevant modality information is utilized. Additionally, considering that the inherent property of multilayer sequential encoding in existing encoders often leads to the loss of low-level information, we adopt a multilevel optimal transport fusion module to preserve low-level features while maintaining multilayer encoding, yielding more expressive representations. Correspondingly, we also create a Unified Multimodal Relation Extraction (UMRE) dataset to evaluate the effectiveness of our framework, encompassing diverse cases where the head and tail entities can originate from either text or image. Extensive experiments show that REMOTE effectively extracts various types of relational triplets and achieves state-of-the-art performanc on almost all metrics across two other public MRE datasets. We release our resources at https://github.com/Nikol-coder/REMOTE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。