用预训练模型将视频转为可解释的文本标签,提升识别准确率。
VTD-CLIP: Video-to-Text Discretization via Prompting CLIP
- 用冻结文本编码器构建视觉代码本,把视频帧转成文本标记。
- 在多个数据集上超越现有方法,最高提升2.1个百分点。
- 适合需要可解释性的视频理解任务,如医疗或安全监控。
视觉-语言模型在视频识别中表现出强大能力,但现有方法依赖参数高效微调,常因时间建模不足导致可解释性差、泛化能力弱。为此,我们提出一种简单有效的视频到文本离散化框架。该方法利用多对一的视觉-文本嵌入对齐特性,将冻结的文本编码器用于从视频类别标签构建视觉代码本,通过特征查找将时序视觉数据转换为文本标记,实现可解释的视频表示。为增强对无关或噪声帧的鲁棒性,引入置信度感知融合模块,动态加权关键帧,依据代码本评估其语义相关性。此外,通过可学习文本提示实现自适应代码本更新。在HMDB-51、UCF-101、SSv2和Kinetics-400上的大量实验验证了该方法的优越性,相较先进方法取得更具竞争力的提升,最高达2.1个百分点。代码将公开于https://github.com/isxinxin/VTD-CLIP。
原文摘要 · Abstract (English)
Vision-language models bridge visual and linguistic understanding and have proven to be powerful for video recognition tasks. Existing approaches primarily rely on parameter-efficient fine-tuning of image-text pre-trained models, yet they often suffer from limited interpretability and poor generalization due to inadequate temporal modeling. To address these, we propose a simple yet effective video-to-text discretization framework. Our method repurposes the frozen text encoder to construct a visual codebook from video class labels due to the many-to-one contrastive alignment between visual and textual embeddings in multimodal pretraining. This codebook effectively transforms temporal visual data into textual tokens via feature lookups and offers interpretable video representations through explicit video modeling. Then, to enhance robustness against irrelevant or noisy frames, we introduce a confidence-aware fusion module that dynamically weights keyframes by assessing their semantic relevance via the codebook. Furthermore, our method incorporates learnable text prompts to conduct adaptive codebook updates. Extensive experiments on HMDB-51, UCF-101, SSv2, and Kinetics-400 have validated the superiority of our approach, achieving more competitive improvements over state-of-the-art methods. The code will be publicly available at https://github.com/isxinxin/VTD-CLIP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。