用时间提示让CLIP模型高效识别动作,少标注也能用。
Is Temporal Prompting All We Need For Limited Labeled Action Recognition?
- 用时间视觉提示替代修改架构,保持CLIP原生能力
- 零样本和少样本下表现超越现有方法,参数少28倍
- 适合资源有限但需快速部署的动作识别场景
视频理解近年取得显著进展,主要依赖大规模标注数据。基于对比预训练的视觉-语言模型在零样本任务中展现出良好泛化能力,有助于减少对标注数据的依赖。现有视频模型通常需修改视觉-语言模型架构以适配视频数据,但这往往计算开销大且难以建模时序信息。本文提出TP-CLIP,一种无需改动核心架构的CLIP适配方法,通过引入时间视觉提示实现时序适应,保留其强泛化能力。该方法高效融入原生CLIP结构,充分利用其预训练优势。在多个数据集上的大量实验表明,其在零样本与少样本学习中均优于现有方法,仅需1/3的GFLOPs和1/28的可调参数,仍能领先15.8%(取决于任务与数据集)。
原文摘要 · Abstract (English)
Video understanding has shown remarkable improvements in recent years, largely dependent on the availability of large scaled labeled datasets. Recent advancements in visual-language models, especially based on contrastive pretraining, have shown remarkable generalization in zero-shot tasks, helping to overcome this dependence on labeled datasets. Adaptations of such models for videos, typically involve modifying the architecture of vision-language models to cater to video data. However, this is not trivial, since such adaptations are mostly computationally intensive and struggle with temporal modeling. We present TP-CLIP, an adaptation of CLIP that leverages temporal visual prompting for temporal adaptation without modifying the core CLIP architecture. This preserves its generalization abilities. TP-CLIP efficiently integrates into the CLIP architecture, leveraging its pre-trained capabilities for video data. Extensive experiments across various datasets demonstrate its efficacy in zero-shot and few-shot learning, outperforming existing approaches with fewer parameters and computational efficiency. In particular, we use just 1/3 the GFLOPs and 1/28 the number of tuneable parameters in comparison to recent state-of-the-art and still outperform it by up to 15.8% depending on the task and dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。