纯文本训练生成图像描述，通过检索增强与模态对齐提升效果
Text-Only Training for Image Captioning with Retrieval Augmentation and Modality Gap Correction
- 用CLIP特征引导语言模型生成，减少图文模态差异
- 结合检索到的句例和潜在向量，使无标注数据也能生成高质量描述
- 无需配对图文数据，适合缺乏标注资源的场景
图像描述任务受到自然语言处理与计算机视觉领域的广泛关注。为降低对人工标注图文对的依赖，已有研究探索了无需任何标注图文对的训练方法，但现有方法仍逊于全监督模型。本文提出TOMCap，一种改进的纯文本训练方法，可在不使用对齐图文对的情况下进行图像描述生成。该方法基于提示预训练语言模型解码器，利用经过模态差距修正后的CLIP表示信息。具体地，我们测试了结合检索到的句例与潜在向量表示来引导生成过程的有效性。大量实验表明，TOMCap优于其他无需训练和纯文本训练方法。我们还分析了检索增强与模态差距修正组件配置对性能的影响。
原文摘要 · Abstract (English)
Image captioning has drawn considerable attention from the natural language processing and computer vision fields. Aiming to reduce the reliance on curated data, several studies have explored image captioning without any humanly-annotated image-text pairs for training, although existing methods are still outperformed by fully supervised approaches. This paper proposes TOMCap, i.e., an improved text-only training method that performs captioning without the need for aligned image-caption pairs. The method is based on prompting a pre-trained language model decoder with information derived from a CLIP representation, after undergoing a process to reduce the modality gap. We specifically tested the combined use of retrieved examples of captions, and latent vector representations, to guide the generation process. Through extensive experiments, we show that TOMCap outperforms other training-free and text-only methods. We also analyze the impact of different choices regarding the configuration of the retrieval-augmentation and modality gap reduction components.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。