用多模态检索增强新闻图像描述,让生成内容更深入
Hierarchical Multi-Modal Retrieval for Knowledge-Grounded News Image Captioning

- 分层多模态检索,结合文本结构与视觉位置匹配
- 在OpenEvent-V1数据集上获0.2824得分,排名第五
- 适合需要背景知识的新闻图像生成任务
传统图像描述方法难以生成富含上下文的完整描述,尤其对非视觉可见的细节。为此,我们提出一种新型检索增强型图像描述框架,通过引入外部知识生成更具洞察力的描述,如物体属性、事件背景和深层意义。该方法采用分层多模态文章检索机制,超越单一文本实体,考虑文章结构感知特征,包括加权文本组件(如标题、正文段落)和视觉布局模式,并结合多维度相似性计算(内容-视觉、视觉-视觉、话语定位)。随后的上下文相关性精炼阶段进一步优化检索信息。检索到的文章作为知识库:首先由视觉语言模型生成简洁图像描述;其次基于该描述提取相关文章信息;最后由大语言模型结合描述与提取知识生成全面、上下文丰富的描述。我们在ACM Multimedia EVENTA 2025挑战赛中以0.2824的总体得分获得第五名,测试集为OpenEvent-V1。源代码已公开于https://github.com/mf0212/EVENTA-Challange。
原文摘要 · Abstract (English)
Traditional image captioning methods often struggle to generate comprehensive, context-rich descriptions, especially for details not directly observable from visual cues. To overcome this, we propose a novel retrieval-augmented image captioning framework that generates captions with deeper insights, such as object attributes, event context, and underlying significance, by leveraging external knowledge. Our approach features a hierarchical multi-modal article retrieval mechanism that moves beyond monolithic text entities. This retrieval considers article structure-aware features, including weighted textual components (e.g., headlines, body sections) and visual placement patterns, alongside multi-faceted similarity computations (content--visual, visual--visual, and discourse positioning). A subsequent contextual relevance refinement stage further enhances the retrieved information. The retrieved articles then serve as the knowledge base for caption generation: first, a VLM generates a concise image description; second, we segment relevant information from the retrieved articles based on this description; and finally, an LLM utilizes both the description and extracted knowledge to generate a comprehensive, contextually detailed caption. We participated in the ACM Multimedia EVENTA 2025 Challenge and achieved 5th place with an overall score of 0.2824 on the private test set of the OpenEvent-V1 dataset. Source code is publicly released at https://github.com/mf0212/EVENTA-Challange.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。