arXiv:2503.15973cs.CV2025-03CVPR被引 14

动态提示让模型更好理解视频,自动聚焦关键帧和区域。

STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understanding

  • 用时空双模块动态生成提示,自适应关注视频中的重要区域和帧。
  • 在多个视频基准上超越现有方法,最高提升4.2%准确率。
  • 适合做视频理解、动作识别的开发者与研究者参考。

预训练于大量图像-文本对的视觉语言模型(如CLIP)在图像任务中展现出出色的零样本泛化能力。然而,将其能力扩展到视频任务仍面临标签数据稀缺和训练成本高的挑战。现有视频提示方法通常使用单一静态提示处理所有视频序列,忽略了帧间的时间动态与空间变化差异,严重影响模型捕捉关键时序信息的能力。为此,本文提出集成时空动态提示(STOP)模型,包含两个互补模块:帧内空间提示与帧间时间提示。帧内空间提示通过利用帧内注意力与时间变化,自适应突出每帧中的判别区域,使模型关注具有显著时间动态的局部细节;帧间时间提示则基于帧相似性度量,在时间方差大的帧之间动态插入提示,强调关键帧的重要性,增强对时序依赖关系的理解。在多个视频基准上的实验表明,STOP持续优于当前最优方法,性能显著提升。

原文摘要 · Abstract (English)

Pre-trained on tremendous image-text pairs, vision-language models like CLIP have demonstrated promising zero-shot generalization across numerous image-based tasks. However, extending these capabilities to video tasks remains challenging due to limited labeled video data and high training costs. Recent video prompting methods attempt to adapt CLIP for video tasks by introducing learnable prompts, but they typically rely on a single static prompt for all video sequences, overlooking the diverse temporal dynamics and spatial variations that exist across frames. This limitation significantly hinders the model's ability to capture essential temporal information for effective video understanding. To address this, we propose an integrated Spatial-TempOral dynamic Prompting (STOP) model which consists of two complementary modules, the intra-frame spatial prompting and inter-frame temporal prompting. Our intra-frame spatial prompts are designed to adaptively highlight discriminative regions within each frame by leveraging intra-frame attention and temporal variation, allowing the model to focus on areas with substantial temporal dynamics and capture fine-grained spatial details. Additionally, to highlight the varying importance of frames for video understanding, we further introduce inter-frame temporal prompts, dynamically inserting prompts between frames with high temporal variance as measured by frame similarity. This enables the model to prioritize key frames and enhances its capacity to understand temporal dependencies across sequences. Extensive experiments on various video benchmarks demonstrate that STOP consistently achieves superior performance against state-of-the-art methods. The code is available at https://github.com/zhoujiahuan1991/CVPR2025-STOP.

视频理解动态提示CLIP时空建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。