用外部文本增强图像描述，让图片说明更丰富有背景。
Beyond Vision: Contextually Enriched Image Captioning with Multi-Modal Retrieval
- 通过多模态检索获取与图像语义相似的图文资料
- 在OpenEvents数据集上生成的描述信息量显著提升
- 适合新闻、教育等需深度理解图像的场景
现实中的图像描述常缺乏上下文深度，遗漏事件背景、时间线索、结果及不可见命名实体等关键信息，限制了其在新闻、教育和数字档案等领域的应用。为此，我们提出一种多模态管道：利用BEIT-3（Flickr30k-384和COCO-384）与SigLIP So-384检索语义相似图像，通过ORB和SIFT进行几何对齐重排，再通过语义搜索从相关文章中提取上下文信息。最后，使用微调后的Qwen3模型（结合QLoRA）融合Instruct BLIP（Vicuna-7B）生成的基础描述，输出事件丰富、上下文感知的描述。在OpenEvents v1数据集上的评估显示，该方法生成的描述显著更富信息量，展现出在需要深层视觉-文本理解的真实场景中的巨大潜力。
原文摘要 · Abstract (English)
Real-world image captions often lack contextual depth, omitting crucial details such as event background, temporal cues, outcomes, and named entities that are not visually discernible. This gap limits the effectiveness of image understanding in domains like journalism, education, and digital archives, where richer, more informative descriptions are essential. To address this, we propose a multimodal pipeline that augments visual input with external textual knowledge. Our system retrieves semantically similar images using BEIT-3 (Flickr30k-384 and COCO-384) and SigLIP So-384, reranks them using ORB and SIFT for geometric alignment, and extracts contextual information from related articles via semantic search. A fine-tuned Qwen3 model with QLoRA then integrates this context with base captions generated by Instruct BLIP (Vicuna-7B) to produce event-enriched, context-aware descriptions. Evaluated on the OpenEvents v1 dataset, our approach generates significantly more informative captions compared to traditional methods, showing strong potential for real-world applications requiring deeper visual-textual understanding
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。