arXiv:2609.04800cs.CV2026-09

让冻结的CLIP模型学会看视频时间动态,精准定位相关片段。

Intrinsic Temporal Adaptation of CLIP for Partially Relevant Video Retrieval

论文配图:Intrinsic Temporal Adaptation of CLIP for Partially Relevant Video Retrieval
图 1 · 摘自论文原文
  • 用轻量适配层让CLIP关注相邻帧组,实现时间感知特征提取。
  • 在MSR-VTT和DiDeMo上达到新最优,帧级匹配精度显著提升。
  • 适合需要细粒度视频检索的研究者,尤其关注跨数据集泛化。

部分相关视频检索(PRVR)旨在从非剪辑视频中找出与文本查询相关的片段。由于目标片段仅占视频一小部分,需超越粗粒度视频级匹配的细粒度理解。现有方法多依赖冻结的CLIP帧特征,缺乏时序建模能力。即使近期出现参数高效适配方法,仍可能依赖不精确的帧级证据进行视频级预测。本文提出内在时序适配(ITA)框架:首先,通过骨干内部时序适配,使最后几层视觉变换器对相邻帧组进行注意力计算,生成时序感知的帧嵌入,同时保持CLIP冻结,仅训练适配参数;其次,引入亲和加权梯度传播机制,基于文本-帧亲和度软聚合前k个最相关帧,并将学习信号传播至多个与查询相关的帧。实验表明,该方法在PRVR基准测试中取得当前最优性能,具备强跨数据集迁移能力,并显著提升真实相关片段内的帧级证据准确性。代码已开源。

原文摘要 · Abstract (English)

Partially Relevant Video Retrieval (PRVR) aims to retrieve untrimmed videos that contain moments relevant to a text query. Since the target moment occupies only a portion of the video, PRVR requires retrieval based on fine-grained understanding beyond coarse video-level matching. However, existing methods often rely on frozen CLIP frame features, which lack temporal understanding. Even with recent progress in parameter-efficient CLIP adaptation, video-level predictions can still be supported by imprecise frame-level evidence. In this paper, we propose an Intrinsic Temporal Adaptation (ITA) framework for PRVR. First, our Backbone-Internal Temporal Adaptation allows the last few visual transformer layers to attend over groups of neighboring frames. This provides temporally aware frame embeddings while keeping CLIP frozen and training only adaptation parameters. Second, we introduce Affinity-Weighted Gradient Propagation to address the weakly supervised nature of PRVR, softly aggregating top-$k$ frames based on text-frame affinities and propagating learning signals to multiple query-relevant frames. Our method achieves state-of-the-art performance on PRVR benchmarks, demonstrates robust cross-dataset transfer, and retrieves substantially more accurate frame-level evidence within ground-truth query-relevant moments. Our code is available at github.com/hynnsk/ITA.

视频检索CLIP时序建模细粒度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。