用代码形式统一处理图文信息抽取,提升结构化结果精度。
Code-MIE: A Code-style Model for Multimodal Information Extraction with Scene Graph and Entity Attribute Knowledge Enhancement
- 将信息抽取任务转化为代码理解与生成,输入为带属性的函数,输出为字典结果。
- 融合实体属性、场景图与视觉特征,在多个数据集上达到60%以上准确率。
- 支持多模态统一框架,适合需要结构化输出的跨模态信息提取场景。
随着大语言模型的发展,基于LLM的信息抽取研究日益增多,但现有方法仍存在改进空间。首先,多数多模态信息抽取(MIE)方法采用自然语言模板作为输入输出,与以实体和关系为主的结构化任务不匹配。其次,虽有少量工作尝试使用代码风格模板,但仅限于纯文本场景,且设计复杂,需为每类任务单独定制。本文提出代码风格多模态信息抽取框架Code-MIE,将MIE统一为代码理解与生成任务。其创新设计包括:(1) 从文本中提取实体属性(如性别、隶属关系),辅助模型理解实体上下文与角色;(2) 将图像转换为场景图与视觉特征,融入丰富视觉信息;(3) 输入模板构造成包含实体属性、场景图与原始文本的Python函数,输出则为包含实体、关系等的Python字典。在M$^3$D、Twitter-15、Twitter-17和MNRE数据集上的实验表明,该方法在英文和中文的M$^3$D数据集上分别达到61.03%和60.49%的准确率,在其余三个数据集上分别取得76.04%、88.07%和73.94%的表现,优于六种基线模型,达到当前最优水平。
原文摘要 · Abstract (English)
With the rapid development of large language models (LLMs), more and more researchers have paid attention to information extraction based on LLMs. However, there are still some spaces to improve in the existing related methods. First, existing multimodal information extraction (MIE) methods usually employ natural language templates as the input and output of LLMs, which mismatch with the characteristics of information tasks that mostly include structured information such as entities and relations. Second, although a few methods have adopted structured and more IE-friendly code-style templates, they just explored their methods on text-only IE rather than multimodal IE. Moreover, their methods are more complex in design, requiring separate templates to be designed for each task. In this paper, we propose a Code-style Multimodal Information Extraction framework (Code-MIE) which formalizes MIE as unified code understanding and generation. Code-MIE has the following novel designs: (1) Entity attributes such as gender, affiliation are extracted from the text to guide the model to understand the context and role of entities. (2) Images are converted into scene graphs and visual features to incorporate rich visual information into the model. (3) The input template is constructed as a Python function, where entity attributes, scene graphs and raw text compose of the function parameters. In contrast, the output template is formalized as Python dictionaries containing all extraction results such as entities, relations, etc. To evaluate Code-MIE, we conducted extensive experiments on the M$^3$D, Twitter-15, Twitter-17, and MNRE datasets. The results show that our method achieves state-of-the-art performance compared to six competing baseline models, with 61.03\% and 60.49\% on the English and Chinese datasets of M$^3$D, and 76.04\%, 88.07\%, and 73.94\% on the other three datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。