通过重复关键词提升视频文本检索精度,无需额外训练。
Repeating Words for Video-Language Retrieval with Coarse-to-Fine Objectives
- 设计粗到细目标函数,增强视频与文本语义对齐。
- 利用关键词重复现象,推理阶段提升召回率2.1%和1.6%。
- 无需预训练,适合资源有限但需高精度检索场景。
视频流的爆炸式增长给视频-文本检索带来了高精度与低训练成本的挑战。现有方法依赖大规模预训练,导致计算开销大,且对视频与文本中的细粒度信息挖掘不足。为此,我们提出一种新框架,通过细粒度特征学习实现更好对齐,并设计无需额外训练的推理流程。具体地,采用粗到细的目标函数(包括对比学习与匹配学习),利用粒度感知表示模块获取细粒度训练数据,该模块基于视频帧与字幕词之间的相似性分析构建。此外,我们发现原始字幕中关键词的重复(称为“重复”)可增强检索性能并改善跨模态对齐。基于此,提出一种包含投票机制与新匹配熵指标的推理管道,在不增加训练的前提下显著提升性能。在四个基准测试上验证,该方法优于以往模型;尤其在MSR-VTT数据集上,Recall@1提升2.1%,DiDeMo上提升1.6%。
原文摘要 · Abstract (English)
The explosive growth of video streaming presents challenges in achieving high accuracy and low training costs for video-language retrieval. However, existing methods rely on large-scale pre-training to improve video retrieval performance, resulting in significant computational demands. Additionally, the fine-grained information in videos and texts remains underexplored. To alleviate these problems, we propose a novel framework to learn fine-grained features for better alignment and introduce an inference pipeline to improve performance without additional training. Specifically, we employ coarse-to-fine objectives to understand the semantic information of video-text pairs, including contrastive and matching learning. The fine-grained data used for training is obtained through the Granularity-Aware Representation module, which is designed based on similarity analysis between video frames and words in captions. Furthermore, we observe that the repetition of keywords in the original captions, referred to as "Repetition", can enhance retrieval performance and improve alignment between video and text. Based on this insight, we propose a novel and effective inference pipeline that incorporates a voting mechanism and a new Matching Entropy metric to achieve better retrieval performance without requiring additional pre-training. Experimental results on four benchmarks demonstrate that the proposed method outperforms previous approaches. Additionally, our inference pipeline achieves significant performance improvements, with a 2.1% increase in Recall@1 on the MSR-VTT dataset and a 1.6% increase on the DiDeMo dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。