通过多轮图文协作反思，提升多模态实体链接准确率
I2CR: Intra- and Inter-modal Collaborative Reflections for Multimodal Entity Linking
- 优先用文本推理，不足时迭代融合图像关键线索
- 在三个数据集上分别提升3.2%、5.1%、1.6%的准确率
- 适合需要高精度多模态匹配的研究与应用
多模态实体链接在众多应用中至关重要。基于大语言模型的方法已成为主流，有效结合文本与视觉信息提升性能。然而，现有方法仍面临两大挑战：在某些场景下无谓引入图像数据，且仅依赖一次视觉特征提取，影响效果与准确性。为此，我们提出一种新型基于大语言模型的框架——跨模态协同反思（I2CR），优先利用文本信息完成任务。当文本不足以准确链接实体时，通过多轮迭代策略，从图像多个维度整合关键视觉线索，支持推理并提升匹配精度。在三个公开数据集上的大量实验表明，该框架持续优于当前最先进方法，在三个数据集上分别取得3.2%、5.1%和1.6%的性能提升。代码已开源：https://github.com/ziyan-xiaoyu/I2CR/
原文摘要 · Abstract (English)
Multimodal entity linking plays a crucial role in a wide range of applications. Recent advances in large language model-based methods have become the dominant paradigm for this task, effectively leveraging both textual and visual modalities to enhance performance. Despite their success, these methods still face two challenges, including unnecessary incorporation of image data in certain scenarios and the reliance only on a one-time extraction of visual features, which can undermine their effectiveness and accuracy. To address these challenges, we propose a novel LLM-based framework for the multimodal entity linking task, called Intra- and Inter-modal Collaborative Reflections. This framework prioritizes leveraging text information to address the task. When text alone is insufficient to link the correct entity through intra- and inter-modality evaluations, it employs a multi-round iterative strategy that integrates key visual clues from various aspects of the image to support reasoning and enhance matching accuracy. Extensive experiments on three widely used public datasets demonstrate that our framework consistently outperforms current state-of-the-art methods in the task, achieving improvements of 3.2%, 5.1%, and 1.6%, respectively. Our code is available at https://github.com/ziyan-xiaoyu/I2CR/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。