提出主动发现视频关键片段的方法,提升文本检索视频的效率与准确率。
Towards Efficient Partially Relevant Video Retrieval with Active Moment Discovering
- 用可学习片段锚点定位语义相关视频段,结合掩码注意力抑制冗余背景。
- 在TVR数据集上参数量减少15.5倍,召回率提升6.0点,性能超越现有方法。
- 适合需要高效精准视频片段检索的应用场景,如智能剪辑与内容摘要。
部分相关视频检索(PRVR)是文本到视频检索中的实际且具有挑战性的任务,视频为未剪裁且包含大量背景内容。目标是实现高效且有效的部分对应关系捕捉。现有方法通常聚焦多尺度片段表示,但存在内容独立性和信息冗余问题,影响检索效果。为此,我们提出一种简单而有效的方法——主动时刻发现网络(AMDNet),致力于发现与查询语义一致的视频片段。通过使用可学习的跨度锚点捕捉不同片段,并应用掩码多时刻注意力机制突出显著片段、抑制冗余背景,实现更紧凑、信息更丰富的视频表示。为进一步增强片段建模,引入片段多样性损失以鼓励不同区域的差异性,以及片段相关性损失以促进与查询语义相关的片段,并与部分相关检索损失联合进行端到端优化。在两个大规模视频数据集(TVR 和 ActivityNet Captions)上的大量实验表明,AMDNet 具有优越性和高效性。特别地,在 TVR 上,其参数量约为最新方法 GMMFormer 的 15.5 倍,但总召回率(SumR)高出 6.0 点。
原文摘要 · Abstract (English)
Partially relevant video retrieval (PRVR) is a practical yet challenging task in text-to-video retrieval, where videos are untrimmed and contain much background content. The pursuit here is of both effective and efficient solutions to capture the partial correspondence between text queries and untrimmed videos. Existing PRVR methods, which typically focus on modeling multi-scale clip representations, however, suffer from content independence and information redundancy, impairing retrieval performance. To overcome these limitations, we propose a simple yet effective approach with active moment discovering (AMDNet). We are committed to discovering video moments that are semantically consistent with their queries. By using learnable span anchors to capture distinct moments and applying masked multi-moment attention to emphasize salient moments while suppressing redundant backgrounds, we achieve more compact and informative video representations. To further enhance moment modeling, we introduce a moment diversity loss to encourage different moments of distinct regions and a moment relevance loss to promote semantically query-relevant moments, which cooperate with a partially relevant retrieval loss for end-to-end optimization. Extensive experiments on two large-scale video datasets (\ie, TVR and ActivityNet Captions) demonstrate the superiority and efficiency of our AMDNet. In particular, AMDNet is about 15.5 times smaller (\#parameters) while 6.0 points higher (SumR) than the up-to-date method GMMFormer on TVR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。