用冻结的图像模型做视频理解，只训练时间模块，省数据省算力。
Towards Data-Efficient Video Pre-training with Frozen Image Foundation Models
- 冻结预训练图像模型作空间编码，只训练递归时间模块。
- 多任务实验显示无需大规模视频预训练也能达到强时序性能。
- 适合想降低视频训练成本的研究者或资源受限团队。
视频基础模型在多项视频理解任务中表现优异，但通常需在大规模视频数据集上进行海量预训练，导致高昂的数据与计算成本。相比之下，现代图像基础模型已具备强大的空间表征能力。这引发一个关键问题：能否通过复用这些空间表征，并仅对时间推理部分进行训练，构建出具有竞争力的视频模型？本文探索一种轻量级训练范式，即冻结预训练图像基础模型，仅训练一个循环时间模块以处理流式视频。通过将图像基础模型作为空间编码器，该方法相比端到端视频预训练可显著减少所需视频数据和计算量。我们在多个视频理解任务上的实证结果表明，即使不进行大规模视频预训练，也能涌现出强大的时序性能，为未来基于冻结图像模型之上预训练时间模块的循环视频基础模型研究提供动力。代码：https://github.com/tue-mps/towards-video-image-frozen。
原文摘要 · Abstract (English)
Video foundation models achieve strong performance across many video understanding tasks, but typically require large-scale pre-training on massive video datasets, resulting in substantial data and compute costs. In contrast, modern image foundation models already provide powerful spatial representations. This raises an important question: can competitive video models be built by reusing these spatial representations and pre-training only for temporal reasoning? We take initial steps toward exploring a lightweight training paradigm that freezes a pre-trained image foundation model and trains only a recurrent temporal module to process streaming video. By reusing an image foundation model as a spatial encoder, this approach could significantly reduce the amount of video data and compute required compared to end-to-end video pre-training. In this work, we explore the feasibility of this approach before investing in computing for video pre-training. Our empirical findings across multiple video understanding tasks suggest that strong temporal performance can emerge without large-scale video pre-training, motivating future work on recurrent video foundation models obtained by pre-training a temporal module on top of a frozen image foundation model. Code: https://github.com/tue-mps/towards-video-image-frozen .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。