通过密集到稀疏特征蒸馏，提升图文匹配的语义表达能力
Aligning Information Capacity Between Vision and Language via Dense-to-Sparse Feature Distillation for Image-Text Matching
- 分两阶段训练：先用密集文本增强视觉表征，再蒸馏稀疏文本
- 在MS-COCO和Flickr30K上优于现有最先进方法
- 适合需要高精度图文对齐的应用场景
让视觉语义模型有效处理多视角描述匹配仍是长期挑战。现有方法通常学习一组嵌入以寻找每视图文本的最优匹配并计算相似度，但这些视觉与文本嵌入的信息容量有限，易受局部相似负样本干扰。为此，我们提出一种名为D2S-VSE的密集到稀疏特征蒸馏视觉语义嵌入方法，通过利用密集文本蒸馏来增强稀疏文本的信息容量。该方法为两阶段框架：预训练阶段，将图像与密集文本对齐以增强视觉语义嵌入的信息容量；微调阶段，同时优化两项任务——将密集文本嵌入蒸馏至稀疏文本嵌入，并对齐图像与稀疏文本，从而提升稀疏文本嵌入的信息容量。所提D2S-VSE模型在大规模MS-COCO和Flickr30K数据集上进行了广泛评估，结果表明其优于近期最先进的方法。
原文摘要 · Abstract (English)
Enabling Visual Semantic Models to effectively handle multi-view description matching has been a longstanding challenge. Existing methods typically learn a set of embeddings to find the optimal match for each view's text and compute similarity. However, the visual and text embeddings learned through these approaches have limited information capacity and are prone to interference from locally similar negative samples. To address this issue, we argue that the information capacity of embeddings is crucial and propose Dense-to-Sparse Feature Distilled Visual Semantic Embedding (D2S-VSE), which enhances the information capacity of sparse text by leveraging dense text distillation. Specifically, D2S-VSE is a two-stage framework. In the pre-training stage, we align images with dense text to enhance the information capacity of visual semantic embeddings. In the fine-tuning stage, we optimize two tasks simultaneously, distilling dense text embeddings to sparse text embeddings while aligning images and sparse texts, enhancing the information capacity of sparse text embeddings. Our proposed D2S-VSE model is extensively evaluated on the large-scale MS-COCO and Flickr30K datasets, demonstrating its superiority over recent state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。