arXiv:2412.07157cs.CV2024-12AAAI被引 12

通过多尺度对比学习,提升视频时序定位中长短期片段的语义表达能力。

Multi-Scale Contrastive Learning for Video Temporal Grounding

  • 利用编码器各层特征自动生成正负样本,无需数据增强或内存库。
  • 在长视频和短视频上均实现显著提升,长视频精度提高2.3%。
  • 适合需要精准时序定位的视频理解任务,如智能检索与内容生成。

时序定位旨在定位与自然语言查询相关的视频片段,是视觉-语言学习与视频理解的核心问题。为编码不同长度的视频片段,现有方法采用称为特征金字塔的多层级结构:低层聚焦短时片段,高层处理长时片段。然而,高层因下采样导致信息容量下降,使片段表示质量受损。为此,本文提出一种对比学习框架,以捕捉视频片段间的显著语义。核心方法是利用视频编码器各阶段特征空间中的样本,无需数据增强或在线记忆库即可获取正负样本。通过设计采样过程,从同一查询中提取多个对应视频片段,并结合其跨层表示,构建一种新型多尺度与跨尺度对比学习机制,将局部短时片段与全局长时片段关联。大量实验表明,该框架在长视频与短视频定位任务中均表现优异,尤其在长视频上达到2.3%的精度提升。代码已开源。

原文摘要 · Abstract (English)

Temporal grounding, which localizes video moments related to a natural language query, is a core problem of vision-language learning and video understanding. To encode video moments of varying lengths, recent methods employ a multi-level structure known as a feature pyramid. In this structure, lower levels concentrate on short-range video moments, while higher levels address long-range moments. Because higher levels experience downsampling to accommodate increasing moment length, their capacity to capture information is reduced and consequently leads to degraded information in moment representations. To resolve this problem, we propose a contrastive learning framework to capture salient semantics among video moments. Our key methodology is to leverage samples from the feature space emanating from multiple stages of the video encoder itself requiring neither data augmentation nor online memory banks to obtain positive and negative samples. To enable such an extension, we introduce a sampling process to draw multiple video moments corresponding to a common query. Subsequently, by utilizing these moments' representations across video encoder layers, we instantiate a novel form of multi-scale and cross-scale contrastive learning that links local short-range video moments with global long-range video moments. Extensive experiments demonstrate the effectiveness of our framework for not only long-form but also short-form video grounding. Code is available at https://github.com/nguyentthong/MSCL

视频理解对比学习时序定位

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。