让模型理解句子整体意义，精准定位视频中对应片段。
Let Me Finish My Sentence: Video Temporal Grounding with Holistic Text Understanding
- 引入帧级门控机制，融合句子整体语义信息
- 在多个基准上超越现有方法，提升定位准确率
- 适合需要精准理解长句的视频检索任务
视频时间定位（VTG）旨在识别与文本查询匹配的视频帧。现有方法通常使用交叉注意力将视觉帧与文本词元序列逐一关联，但忽略了查询句的整体语义理解。模型可能仅关注个别词元与任意帧的局部关联，而忽视全局语义。为此，本文提出两项主要贡献：(1) 帧级门控机制，整合句子层面的文本信息；(2) 跨模态对齐损失，学习查询与相关帧之间的细粒度关联。该方法有效抑制了无关视觉帧的影响，强化了词元的正则化作用。实验表明，所提方法在多个VTG基准上优于当前最优模型，验证了整体语义理解有助于模型聚焦视频中的语义关键区域。
原文摘要 · Abstract (English)
Video Temporal Grounding (VTG) aims to identify visual frames in a video clip that match text queries. Recent studies in VTG employ cross-attention to correlate visual frames and text queries as individual token sequences. However, these approaches overlook a crucial aspect of the problem: a holistic understanding of the query sentence. A model may capture correlations between individual word tokens and arbitrary visual frames while possibly missing out on the global meaning. To address this, we introduce two primary contributions: (1) a visual frame-level gate mechanism that incorporates holistic textual information, (2) cross-modal alignment loss to learn the fine-grained correlation between query and relevant frames. As a result, we regularize the effect of individual word tokens and suppress irrelevant visual frames. We demonstrate that our method outperforms state-of-the-art approaches in VTG benchmarks, indicating that holistic text understanding guides the model to focus on the semantically important parts within the video.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。