用知识增强新闻配图生成，让描述更完整准确
Knowledge Completes the Vision: A Multimodal Entity-aware Retrieval-Augmented Generation Framework for News Image Captioning
- 构建以实体为中心的多模态知识库，融合文本、图像和结构化信息
- 在两个数据集上提升标题质量（CIDEr增6.84/1.16）和实体识别率（F1增4.14/2.64）
- 可泛化到未见数据集，适合新闻生成与跨域应用
新闻图像配图生成旨在结合视觉内容与相关文章的上下文线索，生成具有新闻价值的描述。尽管近期取得进展，现有方法仍面临三大挑战：信息覆盖不全、跨模态对齐弱、视觉实体定位不佳。为此，我们提出MERGE，首个面向新闻图像配图生成的多模态实体感知检索增强生成框架。MERGE构建以实体为中心的多模态知识库（EMKB），整合文本、视觉与结构化知识，实现丰富背景信息检索；通过多阶段假说-标题策略强化跨模态对齐；并基于图像内容动态检索，提升视觉实体匹配精度。在GoodNews与NYTimes800k数据集上的实验表明，MERGE显著优于现有最先进基线，标题质量CIDEr分别提升+6.84和+1.16，命名实体识别F1分数提升+4.14和+2.64。尤其在未见的Visual News数据集上，表现进一步提升，CIDEr增+20.17，F1增+6.22，展现出强鲁棒性与领域适应能力。
原文摘要 · Abstract (English)
News image captioning aims to produce journalistically informative descriptions by combining visual content with contextual cues from associated articles. Despite recent advances, existing methods struggle with three key challenges: (1) incomplete information coverage, (2) weak cross-modal alignment, and (3) suboptimal visual-entity grounding. To address these issues, we introduce MERGE, the first Multimodal Entity-aware Retrieval-augmented GEneration framework for news image captioning. MERGE constructs an entity-centric multimodal knowledge base (EMKB) that integrates textual, visual, and structured knowledge, enabling enriched background retrieval. It improves cross-modal alignment through a multistage hypothesis-caption strategy and enhances visual-entity matching via dynamic retrieval guided by image content. Extensive experiments on GoodNews and NYTimes800k show that MERGE significantly outperforms state-of-the-art baselines, with CIDEr gains of +6.84 and +1.16 in caption quality, and F1-score improvements of +4.14 and +2.64 in named entity recognition. Notably, MERGE also generalizes well to the unseen Visual News dataset, achieving +20.17 in CIDEr and +6.22 in F1-score, demonstrating strong robustness and domain adaptability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。