arXiv:2604.17319cs.CVcs.CL2026-04ACL

端到端生成式多模态命名实体识别,提升图文实体定位准确性

E2E-GMNER: End-to-End Generative Grounded Multimodal Named Entity Recognition

论文配图:E2E-GMNER: End-to-End Generative Grounded Multimodal Named Entity Recognition
图 1 · 摘自论文原文
  • 统一文本识别、类型判断与视觉定位的生成框架
  • 在两个基准上超越现有方法,实体定位准确率显著提升
  • 适合研究多模态理解与大模型应用的开发者

基于多模态命名实体识别(GMNER)旨在联合识别文本中的命名实体提及、预测其语义类型,并将每个实体关联到对应图像中的视觉区域。现有方法多采用流水线架构,分离文本实体识别与视觉定位,导致错误累积和次优联合优化。本文提出E2E-GMNER,一种全端到端的生成式框架,将实体识别、语义分类、视觉定位与隐式知识推理统一于单一多模态大语言模型中。我们将GMNER建模为指令微调的条件生成任务,并引入思维链推理,使模型能自适应判断何时依赖视觉证据或背景知识,降低对噪声信号的依赖。为缓解生成框预测的不稳定性,提出高斯风险感知框扰动(GRBP),以概率扰动的软目标替代硬框监督,增强对标注噪声和离散化误差的鲁棒性。在Twitter-GMNER和Twitter-FMNERG基准上的大量实验表明,E2E-GMNER相较当前最优方法表现优异,验证了统一端到端优化与噪声感知监督的有效性。代码已开源。

原文摘要 · Abstract (English)

Grounded Multimodal Named Entity Recognition (GMNER) aims to jointly identify named entity mentions in text, predict their semantic types, and ground each entity to a corresponding visual region in an associated image. Existing approaches predominantly adopt pipeline-based architectures that decouple textual entity recognition and visual grounding, leading to error accumulation and suboptimal joint optimization. In this paper, we propose E2E-GMNER, a fully end-to-end generative framework that unifies entity recognition, semantic typing, visual grounding, and implicit knowledge reasoning within a single multimodal large language model. We formulate GMNER as an instruction-tuned conditional generation task and incorporate chain-of-thought reasoning to enable the model to adaptively determine when visual evidence or background knowledge is informative, reducing reliance on noisy cues. To further address the instability of generative bounding box prediction, we introduce Gaussian Risk-Aware Box Perturbation (GRBP), which replaces hard box supervision with probabilistically perturbed soft targets to improve robustness against annotation noise and discretization errors. Extensive experiments on the Twitter-GMNER and Twitter-FMNERG benchmarks demonstrate that E2E-GMNER achieves highly competitive performance compared with state of the art methods, validating the effectiveness of unified end-to-end optimization and noise-aware grounding supervision. Code is available at:https://github.com/Finch-coder/E2E-GMNER

多模态命名实体生成式大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。