通过融合视觉语言特征提升视频文本检索效果
DiscoVLA: Discrepancy Reduction in Vision, Language, and Alignment for Parameter-Efficient Video-Text Retrieval
- 融合图像与视频特征，同时解决视觉、语言和对齐三方面差异
- 在MSRVTT数据集上达到50.5% R@1，比之前方法高1.5%
- 适合做视频文本检索的参数高效模型研究者参考
将图像-文本预训练模型CLIP高效适配到视频-文本检索是当前研究热点。尽管CLIP专注于图像级视觉-语言匹配，视频-文本检索需更全面的视频级理解。从图像级迁移到视频级存在三个关键差异：视觉、语言和对齐。现有方法多关注视觉差异，忽视语言和对齐。本文提出DiscoVLA，同时缓解三类差异。引入图像-视频特征融合，整合图像级与视频级特征，有效解决视觉与语言差异；通过生成伪图像描述，学习细粒度图像级对齐；提出图像到视频对齐蒸馏，利用图像级对齐知识增强视频级对齐。大量实验表明，DiscoVLA表现优异，在使用CLIP (ViT-B/16) 的MSRVTT数据集上，R@1达到50.5%，相比之前方法提升1.5%。代码已开源。
原文摘要 · Abstract (English)
The parameter-efficient adaptation of the image-text pretraining model CLIP for video-text retrieval is a prominent area of research. While CLIP is focused on image-level vision-language matching, video-text retrieval demands comprehensive understanding at the video level. Three key discrepancies emerge in the transfer from image-level to video-level: vision, language, and alignment. However, existing methods mainly focus on vision while neglecting language and alignment. In this paper, we propose Discrepancy Reduction in Vision, Language, and Alignment (DiscoVLA), which simultaneously mitigates all three discrepancies. Specifically, we introduce Image-Video Features Fusion to integrate image-level and video-level features, effectively tackling both vision and language discrepancies. Additionally, we generate pseudo image captions to learn fine-grained image-level alignment. To mitigate alignment discrepancies, we propose Image-to-Video Alignment Distillation, which leverages image-level alignment knowledge to enhance video-level alignment. Extensive experiments demonstrate the superiority of our DiscoVLA. In particular, on MSRVTT with CLIP (ViT-B/16), DiscoVLA outperforms previous methods by 1.5% in R@1, reaching a final score of 50.5% R@1. The code is available at https://github.com/LunarShen/DsicoVLA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。