arXiv:2509.17084cs.CV2025-09被引 1

用压缩视频中的运动矢量融合CLIP,实现高效视频识别

MoCLIP-Lite: Efficient Video Recognition by Fusing CLIP with Motion Vectors

  • 冻结CLIP图像编码器,仅训练轻量MLP融合运动矢量特征
  • 在UCF101上达89.2%准确率,远超零样本和仅运动矢量基线
  • 适合追求低计算成本的视频理解应用,如边缘设备部署

视频动作识别是计算机视觉的基础任务,但当前先进模型通常计算开销大且依赖大量视频预训练。与此同时,像对比语言-图像预训练(CLIP)这样的大规模视觉语言模型在静态图像上表现出强大的零样本能力,而运动矢量(MV)则能从压缩视频流中高效提取时间信息。为融合两种范式的优势,我们提出MoCLIP-Lite,一种简单而强大的双流后期融合框架,用于高效视频识别。该方法将冻结的CLIP图像编码器特征与在原始运动矢量上训练的轻量级监督网络特征结合。融合过程中两个主干网络均冻结,仅训练一个微小的多层感知机(MLP)头,确保极高的效率。在UCF101数据集上的全面实验表明,本方法达到89.2%的Top-1准确率,显著优于强零样本(65.0%)和仅运动矢量(66.5%)基线。本工作为视频理解提供了一个高效的新基准,有效弥合了大型静态模型与动态、低成本运动线索之间的差距。代码与模型见https://github.com/microa/MoCLIP-Lite。

原文摘要 · Abstract (English)

Video action recognition is a fundamental task in computer vision, but state-of-the-art models are often computationally expensive and rely on extensive video pre-training. In parallel, large-scale vision-language models like Contrastive Language-Image Pre-training (CLIP) offer powerful zero-shot capabilities on static images, while motion vectors (MV) provide highly efficient temporal information directly from compressed video streams. To synergize the strengths of these paradigms, we propose MoCLIP-Lite, a simple yet powerful two-stream late fusion framework for efficient video recognition. Our approach combines features from a frozen CLIP image encoder with features from a lightweight, supervised network trained on raw MV. During fusion, both backbones are frozen, and only a tiny Multi-Layer Perceptron (MLP) head is trained, ensuring extreme efficiency. Through comprehensive experiments on the UCF101 dataset, our method achieves a remarkable 89.2% Top-1 accuracy, significantly outperforming strong zero-shot (65.0%) and MV-only (66.5%) baselines. Our work provides a new, highly efficient baseline for video understanding that effectively bridges the gap between large static models and dynamic, low-cost motion cues. Our code and models are available at https://github.com/microa/MoCLIP-Lite.

视频识别运动矢量轻量模型零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。