用多模态大模型做实体识别，解决视觉与文本偏见问题。
Beyond Unimodal Shortcuts: MLLMs as Cross-Modal Reasoners for Grounded Named Entity Recognition
- 引入模态感知一致性推理，强制跨模态验证
- 在多个数据集上超越现有基线，提升识别准确率
- 适合研究多模态推理与实体对齐的学者
基于视觉的命名实体识别（GMNER）旨在提取文本实体、赋予语义类别并将其定位到对应视觉区域。本文探索多模态大模型（MLLMs）以端到端方式执行GMNER，突破其作为级联流水线中辅助工具的传统角色。关键发现是：MLLMs存在模态偏见，包括视觉偏见和文本偏见，源于其倾向于采取单模态捷径而非严格的跨模态验证。为此，提出模态感知一致性推理（MCR），通过多风格推理模式注入（MRSI）和约束引导可验证优化（CVO）实现结构化跨模态推理。MRSI将抽象约束转化为可执行推理链，CVO使模型能动态对齐推理路径，并结合组相对策略优化（GRPO）。在GMNER及视觉定位任务上的实验表明，MCR有效缓解模态偏见，性能优于现有基线。
原文摘要 · Abstract (English)
Grounded Multimodal Named Entity Recognition (GMNER) aims to extract text-based entities, assign them semantic categories, and ground them to corresponding visual regions. In this work, we explore the potential of Multimodal Large Language Models (MLLMs) to perform GMNER in an end-to-end manner, moving beyond their typical role as auxiliary tools within cascaded pipelines. Crucially, our investigation reveals a fundamental challenge: MLLMs exhibit $\textbf{modality bias}$, including visual bias and textual bias, which stems from their tendency to take unimodal shortcuts rather than rigorous cross-modal verification. To address this, we propose Modality-aware Consistency Reasoning ($\textbf{MCR}$), which enforces structured cross-modal reasoning through Multi-style Reasoning Schema Injection (MRSI) and Constraint-guided Verifiable Optimization (CVO). MRSI transforms abstract constraints into executable reasoning chains, while CVO empowers the model to dynamically align its reasoning trajectories with Group Relative Policy Optimization (GRPO). Experiments on GMNER and visual grounding tasks demonstrate that MCR effectively mitigates modality bias and achieves superior performance compared to existing baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。