SMILE通过融合空间与运动语义，提升视频自监督学习效果。
SMILE: Infusing Spatial and Motion Semantics in Masked Video Learning
- 用CLIP引导学习，注入高层空间语义信息
- 引入合成运动模式，增强对动态内容的建模能力
- 无需自然视频数据，适合通用视频表征学习
掩码视频建模（如VideoMAE）是有效的视频自监督学习范式，但主要依赖在具有大量时间冗余的自然视频上重建像素级细节，限制了其语义表征能力和运动动态编码。为此，本文提出一种新方法SMILE，通过融合空间与运动语义来改进视频表示学习。SMILE利用图像-语言预训练模型（如CLIP）提供高层空间语义指导，并通过在训练数据中引入合成运动模式，增强模型对复杂动态内容的捕捉能力。此外，基于SMILE可构建一种无需自然视频数据的新型自监督视频学习范式，能够学习强视频表示。我们在7个数据集上进行了广泛实验，涵盖多种下游任务。SMILE显著超越现有最先进方法，在学习更具判别性和泛化性的视频表示方面表现出色。代码已公开：https://github.com/fmthoker/SMILE
原文摘要 · Abstract (English)
Masked video modeling, such as VideoMAE, is an effective paradigm for video self-supervised learning (SSL). However, they are primarily based on reconstructing pixel-level details on natural videos which have substantial temporal redundancy, limiting their capability for semantic representation and sufficient encoding of motion dynamics. To address these issues, this paper introduces a novel SSL approach for video representation learning, dubbed as SMILE, by infusing both spatial and motion semantics. In SMILE, we leverage image-language pretrained models, such as CLIP, to guide the learning process with their high-level spatial semantics. We enhance the representation of motion by introducing synthetic motion patterns in the training data, allowing the model to capture more complex and dynamic content. Furthermore, using SMILE, we establish a new self-supervised video learning paradigm capable of learning strong video representations without requiring any natural video data. We have carried out extensive experiments on 7 datasets with various downstream scenarios. SMILE surpasses current state-of-the-art SSL methods, showcasing its effectiveness in learning more discriminative and generalizable video representations. Code is available: https://github.com/fmthoker/SMILE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。