arXiv:2509.01259cs.CV2025-09被引 2

让图片描述融入事件背景,生成更真实有据的新闻级说明。

ReCap: Event-Aware Image Captioning with Article Retrieval and Semantic Gaussian Normalization

  • 用文章检索+语义归一化,结合新闻内容增强图片描述。
  • 在OpenEvents数据集上取得0.54666的高分,排名第二。
  • 适合新闻、档案等需准确上下文理解的场景。

图像描述系统常产生泛化表达,难以捕捉新闻报道和数字存档中至关重要的事件级语义。本文提出ReCap,一种融合相关文章上下文信息的事件增强型图像检索与描述生成新框架,旨在生成叙事性强且事实准确的描述。该方法克服了传统视觉语言模型仅关注可见内容而忽略时间、社会与历史背景的局限。ReCap包含三个组件:(1) 基于DINOv2嵌入的两级文章检索系统,先用全局特征相似性筛选候选,再通过局部块级互近邻相似性重排序;(2) 从文章摘要、通用描述和原始元数据中综合提取上下文信息;(3) 基于大语言模型的描述生成系统,采用语义高斯归一化提升流畅性与相关性。在EVENTA 2025 Grand Challenge Track 1的OpenEvents V1数据集上评估,ReCap获得0.54666的总分,在私有测试集上排名第二。结果表明其有效连接视觉感知与现实知识,为高风险领域提供可落地的上下文感知图像理解方案。代码已公开于https://github.com/Noridom1/EVENTA2025-Event-Enriched-Image-Captioning。

原文摘要 · Abstract (English)

Image captioning systems often produce generic descriptions that fail to capture event-level semantics which are crucial for applications like news reporting and digital archiving. We present ReCap, a novel pipeline for event-enriched image retrieval and captioning that incorporates broader contextual information from relevant articles to generate narrative-rich, factually grounded captions. Our approach addresses the limitations of standard vision-language models that typically focus on visible content while missing temporal, social, and historical contexts. ReCap comprises three integrated components: (1) a robust two-stage article retrieval system using DINOv2 embeddings with global feature similarity for initial candidate selection followed by patch-level mutual nearest neighbor similarity re-ranking; (2) a context extraction framework that synthesizes information from article summaries, generic captions, and original source metadata; and (3) a large language model-based caption generation system with Semantic Gaussian Normalization to enhance fluency and relevance. Evaluated on the OpenEvents V1 dataset as part of Track 1 in the EVENTA 2025 Grand Challenge, ReCap achieved a strong overall score of 0.54666, ranking 2nd on the private test set. These results highlight ReCap's effectiveness in bridging visual perception with real-world knowledge, offering a practical solution for context-aware image understanding in high-stakes domains. The code is available at https://github.com/Noridom1/EVENTA2025-Event-Enriched-Image-Captioning.

图像描述事件理解上下文增强大模型应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。