arXiv:2605.20713cs.CVcs.AI2026-05

只在必要时调用视觉证据,提升多模态信息抽取的准确率与效率。

SAVER: Selective As-Needed Vision Evidence for Multimodal Information Extraction

论文配图:SAVER: Selective As-Needed Vision Evidence for Multimodal Information Extraction
图 1 · 摘自论文原文
  • 按需激活视觉信息,通过可信度门控判断是否需要图像支持。
  • 在多个数据集上比纯文本和全程融合模型提升F1,同时降低计算开销。
  • 适合处理社交媒体中图文不一致或冗余的复杂场景,适合资源受限部署。

社交媒体中的多模态信息抽取困难,因帖子可能附带多个与文本弱相关、冗余甚至误导的图像。始终启用多模态融合会浪费计算资源,并放大虚假视觉线索。核心挑战在于:对每个候选实体或实体对,判断是否应调用视觉信息,若需,则选择能提供可信证据的少量图像。我们提出SAVER,一种面向多模态命名实体识别(MNER)和关系抽取(MRE)的按需视觉选择框架。SAVER采用一致可验证性门控(CGG)评估MNER中片段级视觉可验证性,从两个标记实体推导出MRE中的对级激活,并通过类似置信区间的方法(Clopper-Pearson上界)在预留数据集上校准激活阈值。激活后,基于子模函数的相关性-多样性选择器从多图中选出紧凑证据子集,再由集合变换器聚合。一个能量启发的联合评分头整合文本、可选视觉证据、图文一致性及稀疏路由信号,用于实体类型分类或关系判别。实验表明,SAVER在多个基准上持续提升F1,优于强基线文本模型和全程融合模型,同时降低AURC,固定风险水平下提高激活覆盖率,并减少FLOPs与P90延迟。

原文摘要 · Abstract (English)

Multimodal IE in social media is difficult because a post may attach multiple images that are weakly related, redundant, or even misleading with respect to the text. In this setting, always-on multimodal fusion wastes computation and can amplify spurious visual cues. The core challenge is to decide, for each candidate span or marked entity pair, whether vision should be consulted at all and, if so, which small subset of images provides trustworthy evidence. We propose SAVER, a selective vision-as-needed framework for multimodal named entity recognition and multimodal relation extraction. SAVER uses a Conformal Groundability Gate (CGG) to estimate span-level visual groundability in MNER, derive pair-level activation in MRE from the two marked entities, and calibrate the activation threshold on a held-out split via a conformal-style procedure with Clopper--Pearson upper bounds. When activated, a submodular relevance--diversity selector chooses a compact evidence subset across images, which is then aggregated by a Set Transformer. An energy-inspired joint scoring head combines text, optional visual evidence, text--image consistency, and sparse routing for entity typing or relation classification. Experiments show that SAVER consistently improves F1 over strong text-only and always-on multimodal baselines, while reducing AURC, increasing activation coverage at a fixed risk level, and lowering FLOPs and P90 latency.

多模态信息抽取按需计算视觉推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。