无需训练即可精准定位视频中文字描述的片段,提升跨域适应能力。
DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding

- 双侧增强:视觉侧融合多尺度时序相似性,文本侧动态优化查询嵌入。
- 在Charades-STA上比最强无训练方法提升5.61点mIoU,跨域测试达50.86。
- 适合追求高效部署、低标注成本的视频定位场景,尤其适用于分布外数据。
文本引导的视频时序定位旨在根据文本查询定位未剪辑视频中的相关片段,但密集时序标注和训练专用模型成本高且在分布外场景下表现脆弱。近期无训练方法通过直接匹配预训练视觉-语言表示缓解此问题,但仍存在两个根本瓶颈:帧级视觉编码忽略时序动态,固定查询嵌入无法解决查询歧义。为此,我们提出DSE-VTG——一种无需任务特定训练的双侧增强框架。视觉侧采用多尺度相似性融合(MSF),将帧级与片段级相似性整合为统一的时序感知相似性图谱;文本侧引入查询级测试时自适应(Q-TTA),通过轻量级加性偏移优化查询嵌入以适配视频,不微调主干或调用外部大模型。在三个标准数据集及两个分布外基准上的实验表明,DSE-VTG在无训练方法中达到最优性能:在Charades-STA上,相比最强前序无训练方法提升5.61点mIoU;在分布外设置下,Charades-CG Novel-Word上达到50.86 mIoU,超越最强监督基线2.76 mIoU。代码将在录用后公开。
原文摘要 · Abstract (English)
Text-guided Video Temporal Grounding (VTG) aims to localize the relevant segments in an untrimmed video based on text queries, yet collecting dense temporal annotations and training task-specific models remain costly and brittle under distribution shift. Recent training-free VTG approaches mitigate this issue by directly matching pretrained vision-language representations, but they still face two fundamental information bottlenecks: frame-wise visual encoding overlooks temporal dynamics, while fixed query embeddings cannot resolve query ambiguity. To address these issues, we propose DSE-VTG, a \underline{D}ual-\underline{S}ide \underline{E}nhancement framework that addresses both without any task-specific training. On the visual side, Multi-scale Similarity Fusion (MSF) combines frame- and clip-level similarities into a unified, temporally aware similarity profile. On the textual side, Query-level Test-Time Adaptation (Q-TTA) optimizes a lightweight additive offset to adapt the query embedding to the video at test time, without finetuning the backbone or calling external large language models. Extensive experiments on three standard and two OOD benchmarks show that DSE-VTG achieves state-of-the-art performance among training-free methods. On Charades-STA, it improves mIoU over the strongest prior training-free method by 5.61 points. Under distribution shift, DSE-VTG reaches 50.86 mIoU on Charades-CG Novel-Word, surpassing the strongest supervised baseline by 2.76 mIoU. Our code will be released upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。