通过分层特征与自适应评分提升视频时间定位精度。
FlashVTG: Feature Layering and Adaptive Score Handling Network for Video Temporal Grounding
- 引入时序特征分层模块,捕捉多尺度视频内容变化。
- 在QVHighlights上短片段定位mAP提升至SOTA的125%。
- 无需额外训练开销,适合实际部署的视频理解任务。
文本引导的视频时间定位(VTG)旨在基于文本描述定位未剪辑视频中的相关片段,包含两个子任务:时刻检索(MR)和亮点检测(HD)。尽管以往方法已取得良好成果,但短视频段的精准定位仍具挑战性,主要源于稀疏且有限的解码器查询,严重制约预测准确性。此外,因仅依赖孤立预测进行排序,忽视视频整体上下文,常导致次优结果。为此,我们提出FlashVTG框架,包含时序特征分层(TFL)模块与自适应评分精炼(ASR)模块。TFL模块替代传统解码结构,以捕捉多时序尺度下的细微内容变化;ASR模块则通过融合相邻时刻与多尺度特征上下文,优化预测排序。大量实验表明,FlashVTG在四个主流数据集上均达当前最优性能。尤其在QVHighlights数据集上,MR的mAP提升5.8%,HD提升3.3%。对于短片段检索,其mAP达到先前SOTA的125%。所有改进均未增加训练负担,凸显其高效性。代码已开源:https://github.com/Zhuo-Cao/FlashVTG。
原文摘要 · Abstract (English)
Text-guided Video Temporal Grounding (VTG) aims to localize relevant segments in untrimmed videos based on textual descriptions, encompassing two subtasks: Moment Retrieval (MR) and Highlight Detection (HD). Although previous typical methods have achieved commendable results, it is still challenging to retrieve short video moments. This is primarily due to the reliance on sparse and limited decoder queries, which significantly constrain the accuracy of predictions. Furthermore, suboptimal outcomes often arise because previous methods rank predictions based on isolated predictions, neglecting the broader video context. To tackle these issues, we introduce FlashVTG, a framework featuring a Temporal Feature Layering (TFL) module and an Adaptive Score Refinement (ASR) module. The TFL module replaces the traditional decoder structure to capture nuanced video content variations across multiple temporal scales, while the ASR module improves prediction ranking by integrating context from adjacent moments and multi-temporal-scale features. Extensive experiments demonstrate that FlashVTG achieves state-of-the-art performance on four widely adopted datasets in both MR and HD. Specifically, on the QVHighlights dataset, it boosts mAP by 5.8% for MR and 3.3% for HD. For short-moment retrieval, FlashVTG increases mAP to 125% of previous SOTA performance. All these improvements are made without adding training burdens, underscoring its effectiveness. Our code is available at https://github.com/Zhuo-Cao/FlashVTG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。