用预训练图像模型加速视频自监督学习，大幅降本提效。
Advancing Video Self-Supervised Learning via Image Foundation Models
- 将图像基础模型与3D残差网络结合，构建视频表征模型。
- 在UCF101上实现顶尖性能，训练时间缩短3.4倍，显存降低8.2倍。
- 适合想低成本训练视频模型的研究者与工程团队。
过去十年，图像基础模型（IFMs）取得了前所未有的进展。然而，直接利用预训练的IFMs进行视频自监督表示学习的潜力仍被严重忽视。本文提出一种名为AdViSe的先进视频自监督学习方法，旨在显著降低视频表征模型的训练开销。具体而言，我们首先在IFMs中引入时序建模模块（ResNet3D），构建视频表示模型；随后采用播放速率感知的自监督学习策略，在冻结IFM组件的同时训练时序模块。在UCF101上的实验表明，AdViSe在性能上媲美现有最先进方法，同时训练时间减少3.4倍，GPU内存使用量降低8.2倍。该研究为基于预训练图像基础模型的低成本视频自监督学习提供了新思路。代码已开源：https://github.com/JingwWu/advise-video-ssl。
原文摘要 · Abstract (English)
In the past decade, image foundation models (IFMs) have achieved unprecedented progress. However, the potential of directly using IFMs for video self-supervised representation learning has largely been overlooked. In this study, we propose an advancing video self-supervised learning (AdViSe) approach, aimed at significantly reducing the training overhead of video representation models using pre-trained IFMs. Specifically, we first introduce temporal modeling modules (ResNet3D) to IFMs, constructing a video representation model. We then employ a video self-supervised learning approach, playback rate perception, to train temporal modules while freezing the IFM components. Experiments on UCF101 demonstrate that AdViSe achieves performance comparable to state-of-the-art methods while reducing training time by $3.4\times$ and GPU memory usage by $8.2\times$. This study offers fresh insights into low-cost video self-supervised learning based on pre-trained IFMs. Code is available at https://github.com/JingwWu/advise-video-ssl.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。