提出新型轻量级视频时间定位方法，显著提升效率与精度。
Sparse-Dense Side-Tuner for efficient Video Temporal Grounding
- 设计无锚点的稀疏-稠密侧调制结构，聚焦关键帧优化。
- 在多个数据集上达到顶尖性能，参数量减少73%。
- 适用于资源受限场景，适合追求高效模型的研究者。
视频时间定位（VTG）包括基于文本查询的时段检索（MR）和亮点检测（HD）。现有方法多依赖冻结大模型的最终层特征，难以适应新领域。全量微调不现实，而参数高效微调（尤其是侧调制，ST）成为有效替代方案。但以往的ST方法从帧级优化出发，忽略了MR的固有稀疏性。为此，我们提出首个面向VTG的无锚点侧调制架构——稀疏-稠密侧调制器（SDST），并引入基于参考的可变形自注意力机制，增强可变形注意力在无锚点方法中的上下文建模能力。此外，首次将InternVideo2骨干网络有效整合进ST框架，展现出显著性能提升。整体方法在QVHighlights、TACoS和Charades-STA上实现高度竞争力或当前最优表现，相比现有SOTA方法参数量减少最高达73%。代码已开源：https://github.com/davidpujol/SDST。
原文摘要 · Abstract (English)
Video Temporal Grounding (VTG) involves Moment Retrieval (MR) and Highlight Detection (HD) based on textual queries. For this, most methods rely solely on final-layer features of frozen large pre-trained backbones, limiting their adaptability to new domains. While full fine-tuning is often impractical, parameter-efficient fine-tuning -- and particularly side-tuning (ST) -- has emerged as an effective alternative. However, prior ST approaches this problem from a frame-level refinement perspective, overlooking the inherent sparse nature of MR. To address this, we propose the Sparse-Dense Side-Tuner (SDST), the first anchor-free ST architecture for VTG. We also introduce the Reference-based Deformable Self-Attention, a novel mechanism that enhances the context modeling of the deformable attention -- a key limitation of existing anchor-free methods. Additionally, we present the first effective integration of InternVideo2 backbone into an ST framework, showing its profound implications in performance. Overall, our method significantly improves existing ST methods, achieving highly competitive or SOTA results on QVHighlights, TACoS, and Charades-STA, while reducing up to a 73% the parameter count w.r.t. the existing SOTA methods. The code is publicly accessible at https://github.com/davidpujol/SDST.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。