让视频句子定位模型全程可训练,提升定位精度。
A Paradigm Shift: Fully End-to-End Training for Temporal Sentence Grounding in Videos
- 端到端联合优化视频主干网络和定位头
- 在两个数据集上超越当前最佳方法
- 适配深层网络,节省内存且增强视觉表征
视频中的句子定位(TSGV)旨在从非剪辑视频中定位与句子查询语义对应的时间片段。现有方法多采用预训练的、与查询无关的视觉编码器进行离线特征提取,视频主干网络被冻结且未针对TSGV任务优化,导致视觉分类训练与任务需求存在偏差。为解决该问题,本文提出全端到端训练范式,联合优化视频主干网络与定位头。通过实证研究验证了端到端学习在不同模型规模下的有效性。进一步提出句式条件适配器(SCADA),利用句子特征自适应地训练视频主干网络的一小部分参数。SCADA通过精确融合语言嵌入,调节特征图,显著提升视觉表示能力,同时支持更深网络部署并降低内存消耗。在两个基准数据集上的实验表明,该方法优于当前最优方法。代码与模型将公开。
原文摘要 · Abstract (English)
Temporal sentence grounding in videos (TSGV) aims to localize a temporal segment that semantically corresponds to a sentence query from an untrimmed video. Most current methods adopt pre-trained query-agnostic visual encoders for offline feature extraction, and the video backbones are frozen and not optimized for TSGV. This leads to a task discrepancy issue for the video backbone trained for visual classification, but utilized for TSGV. To bridge this gap, we propose a fully end-to-end paradigm that jointly optimizes the video backbone and localization head. We first conduct an empirical study validating the effectiveness of end-to-end learning over frozen baselines across different model scales. Furthermore, we introduce a Sentence Conditioned Adapter (SCADA), which leverages sentence features to train a small portion of video backbone parameters adaptively. SCADA facilitates the deployment of deeper network backbones with reduced memory and significantly enhances visual representation by modulating feature maps through precise integration of linguistic embeddings. Experiments on two benchmarks show that our method outperforms state-of-the-art approaches. The code and models will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。