用更少片段高效定位视频中与语言查询相关的片段,提升检索精度。
Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language

- 基于语言查询动态筛选候选视频片段,避免固定长度切分导致的信息丢失。
- 在三个数据集上优于现有方法,显著减少计算量且保持高准确率。
- 可作为插件模块适配主流模型,适合长视频场景的快速检索应用。
给定一段未剪辑的长视频和一句语言查询,语言驱动的视频片段检索(VMR)旨在定位与查询相关的目标片段。由于原始视频过长,现有方法通常将视频稀疏采样为多个固定长度的片段,再与查询特征进行多模态交互推理,这对长达数小时的真实视频不适用。固定长度切分会遗漏部分相关帧,模糊目标片段边界,导致相邻无关帧被误判为边界,引发跨模态错位,产生边界偏差与推理偏差。为此,本文提出高效剪裁方法SpotVMR,能识别与查询相关的视频区域并动态提取关键片段。该方法设计了一种新型片段搜索模型,根据语言查询学习有潜力的视频区域;引入低成本语义索引特征,捕捉物体与交互上下文以指导搜索位置;并通过蒸馏损失缓解端到端联合训练中的优化问题。在三个挑战性数据集上的大量实验验证了其有效性。
原文摘要 · Abstract (English)
Given an untrimmed video and a sentence query, video moment retrieval using language (VMR) aims to locate a target query-relevant moment. Since the untrimmed video is overlong, almost all existing VMR methods first sparsely down-sample each untrimmed video into multiple fixed-length video clips and then conduct multi-modal interactions with the query feature and expensive clip features for reasoning, which is infeasible for long real-world videos that span hours. Since the video is downsampled into fixed-length clips, some query-related frames may be filtered out, which will blur the specific boundary of the target moment, take the adjacent irrelevant frames as new boundaries, easily leading to cross-modal misalignment and introducing both boundary-bias and reasoning-bias. To this end, in this paper, we propose an efficient approach, SpotVMR, to trim the query-relevant clip. Besides, our proposed SpotVMR can serve as plug-and-play module, which achieves efficiency for state-of-the-art VMR methods while maintaining good retrieval performance. Especially, we first design a novel clip search model that learns to identify promising video regions to search conditioned on the language query. Then, we introduce a set of low-cost semantic indexing features to capture the context of objects and interactions that suggest where to search the query-relevant moment. Also, the distillation loss is utilized to address the optimization issues arising from end-to-end joint training of the clip selector and VMR model. Extensive experiments on three challenging datasets demonstrate its effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。