让图片描述更独特:通过对比组内图像差异,突出唯一特征。
Group-based Distinctive Image Captioning with Memory Difference Encoding and Attention
- 构建图像组内差异注意力模块,识别组内独有的物体特征。
- 在多个基线模型上显著提升描述独特性,准确率未明显下降。
- 提出新评估指标DisWordRate,量化描述的独特程度。
当前图像描述研究主要依赖大规模数据与模型提升准确性,虽在BLEU、CIDEr、SPICE等指标上表现优异,但对描述区分目标图像与其他相似图像的能力关注不足。现有方法如对比学习或重加权真值描述,常忽略相似图像组中物体间的关联(如同相册物品或细粒度事件)。本文提出一种基于图像组的差异化描述方法(GDMA),通过视觉比较同一组内图像,突出每张图像的独特性。核心是引入组内差异记忆注意力模块(GDMA),识别在组内其他图像中不常见的物体特征(即低相似性特征),并在生成描述时优先利用这些特征。同时,从真值描述中选取独特词汇,引导语言解码器和GDMA模块。此外,提出新的评估指标Distinctive Word Rate(DisWordRate)量化描述独特性。实验表明,该方法显著提升多个基线模型的描述独特性,且在保持高准确率的同时达到当前最优性能。
原文摘要 · Abstract (English)
Recent advances in image captioning have focused on enhancing accuracy by substantially increasing the dataset and model size. While conventional captioning models exhibit high performance on established metrics such as BLEU, CIDEr, and SPICE, the capability of captions to distinguish the target image from other similar images is under-explored. To generate distinctive captions, a few pioneers employed contrastive learning or re-weighted the ground-truth captions. However, these approaches often overlook the relationships among objects in a similar image group (e.g., items or properties within the same album or fine-grained events). In this paper, we introduce a novel approach to enhance the distinctiveness of image captions, namely Group-based Differential Distinctive Captioning Method, which visually compares each image with other images in one similar group and highlights the uniqueness of each image. In particular, we introduce a Group-based Differential Memory Attention (GDMA) module, designed to identify and emphasize object features in an image that are uniquely distinguishable within its image group, i.e., those exhibiting low similarity with objects in other images. This mechanism ensures that such unique object features are prioritized during caption generation for the image, thereby enhancing the distinctiveness of the resulting captions. To further refine this process, we select distinctive words from the ground-truth captions to guide both the language decoder and the GDMA module. Additionally, we propose a new evaluation metric, the Distinctive Word Rate (DisWordRate), to quantitatively assess caption distinctiveness. Quantitative results indicate that the proposed method significantly improves the distinctiveness of several baseline models, and achieves state-of-the-art performance on distinctiveness while not excessively sacrificing accuracy...
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。