arXiv:2412.01115cs.CV2024-12被引 2

用扩散模型提升图像检索,让生成更懂视觉细节。

DIR: Retrieval-Augmented Image Captioning with Comprehensive Understanding

论文配图:DIR: Retrieval-Augmented Image Captioning with Comprehensive Understanding
图 1 · 摘自论文原文
  • 用扩散模型重建噪声图像,学习更全面的视觉特征
  • 构建高质量检索库,显著提升跨领域生成效果
  • 无需增加计算成本,适合实际部署场景

图像描述模型在新数据集上常因训练数据领域局限而性能下降。现有检索增强方法面临两大挑战:一是用于检索的图像特征基于真实标注文本优化,受限于标注视角与人为偏见;二是对检索到的文本利用不足,仅依赖原始描述或解析对象,难以捕捉完整语义。本文提出Dive Into Retrieval(DIR),通过两项创新提升视觉理解:(1) 扩散引导的检索增强,利用预训练扩散模型重建噪声图像,使图像特征学习更全面、精细的视觉信息;(2) 构建高质量检索数据库,提供丰富语义支持,尤其在跨领域场景下表现优异。大量实验表明,DIR不仅保持了域内竞争力,还显著提升域外泛化能力,且推理开销不变。

原文摘要 · Abstract (English)

Image captioning models often suffer from performance degradation when applied to novel datasets, as they are typically trained on domain-specific data. To enhance generalization in out-of-domain scenarios, retrieval-augmented approaches have garnered increasing attention. However, current methods face two key challenges: (1) image features used for retrieval are often optimized based on ground-truth (GT) captions, which represent the image from a specific perspective and are influenced by annotator biases, and (2) they underutilize the full potential of retrieved text, typically relying on raw captions or parsed objects, which fail to capture the full semantic richness of the data. In this paper, we propose Dive Into Retrieval (DIR), a method designed to enhance both the image-to-text retrieval process and the utilization of retrieved text to achieve a more comprehensive understanding of the visual content. Our approach introduces two key innovations: (1) diffusion-guided retrieval enhancement, where a pretrained diffusion model guides image feature learning by reconstructing noisy images, allowing the model to capture more comprehensive and fine-grained visual information beyond standard annotated captions; and (2) a high-quality retrieval database, which provides comprehensive semantic information to enhance caption generation, especially in out-of-domain scenarios. Extensive experiments demonstrate that DIR not only maintains competitive in-domain performance but also significantly improves out-of-domain generalization, all without increasing inference costs.

图像描述检索增强扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。