用检索生成对象关系提示,训练快、效果好,适合高效图像描述生成。
RORPCap: Retrieval-based Objects and Relations Prompt for Image Captioning
- 基于检索构建对象与关系提示模板,避免冗余检测和复杂图网络
- 仅用2.6小时训练即达120.5% CIDEr和22.0% SPICE得分
- 适合作为轻量高效图像描述生成的替代方案
图像描述任务旨在以开放式方式生成输入图像的自然语言描述。准确生成与图像相关的描述,关键在于识别图像中的对象并理解其关系。现有方法通常依赖目标检测器或结合检测器与图卷积网络(GCN),但面临检测信息冗余、GCN构建困难及训练成本高等问题。为此,本文提出一种基于检索的对象与关系提示方法(RORPCap),受图像-文本检索可提供丰富语义信息的启发。RORPCap首先通过对象与关系提取模型从图像中提取对象和关系词汇,将其嵌入预设提示模板并编码为提示嵌入;随后设计基于Mamba的映射网络,快速将CLIP提取的图像嵌入映射为视觉-文本嵌入;最后将提示嵌入与视觉-文本嵌入拼接,形成富含文本信息的特征嵌入,输入GPT-2模型生成描述。在广泛使用的MS-COCO数据集上,RORPCap仅需2.6小时交叉熵损失训练,即在Karpathy测试集上达到120.5% CIDEr和22.0% SPICE得分,性能媲美检测器与GCN基模型,且训练时间最短,展现出作为图像描述生成替代方案的潜力。
原文摘要 · Abstract (English)
Image captioning aims to generate natural language descriptions for input images in an open-form manner. To accurately generate descriptions related to the image, a critical step in image captioning is to identify objects and understand their relations within the image. Modern approaches typically capitalize on object detectors or combine detectors with Graph Convolutional Network (GCN). However, these models suffer from redundant detection information, difficulty in GCN construction, and high training costs. To address these issues, a Retrieval-based Objects and Relations Prompt for Image Captioning (RORPCap) is proposed, inspired by the fact that image-text retrieval can provide rich semantic information for input images. RORPCap employs an Objects and relations Extraction Model to extract object and relation words from the image. These words are then incorporate into predefined prompt templates and encoded as prompt embeddings. Next, a Mamba-based mapping network is designed to quickly map image embeddings extracted by CLIP to visual-text embeddings. Finally, the resulting prompt embeddings and visual-text embeddings are concatenated to form textual-enriched feature embeddings, which are fed into a GPT-2 model for caption generation. Extensive experiments conducted on the widely used MS-COCO dataset show that the RORPCap requires only 2.6 hours under cross-entropy loss training, achieving 120.5% CIDEr score and 22.0% SPICE score on the "Karpathy" test split. RORPCap achieves comparable performance metrics to detector-based and GCN-based models with the shortest training time and demonstrates its potential as an alternative for image captioning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。