用合成视频和静态图像训练视频模型，性能接近真实视频预训练。
Learning Video Representations without Natural Videos
- 通过简单生成过程构建逐步复杂化的合成视频数据集
- 在UCF101上达到自然视频预训练97.2%的性能，超越HMDB51基准
- 引入静态图像可提升泛化能力，适合追求可控数据的开发者
我们证明，仅使用合成视频和自然图像即可学习有效的视频表示，无需使用真实视频进行训练。提出一系列由简单生成过程构建的视频数据集，逐步模拟自然视频的特性（如运动、加速度、形状变化）。随着数据集复杂度提升，下游视频模型性能持续增强。基于合成视频预训练的VideoMAE模型，在UCF101动作分类任务上，弥补了从零训练与自监督预训练之间97.2%的性能差距，并在HMDB51上表现更优。在预训练中加入静态图像裁剪，性能接近UCF101预训练，且在UCF101-P的14个分布外数据集中有11个表现更优。分析发现帧多样性与与自然数据的相似性与下游性能正相关。该方法为预训练数据构建提供了更可控、透明的替代方案。
原文摘要 · Abstract (English)
We show that useful video representations can be learned from synthetic videos and natural images, without incorporating natural videos in the training. We propose a progression of video datasets synthesized by simple generative processes, that model a growing set of natural video properties (e.g., motion, acceleration, and shape transformations). The downstream performance of video models pre-trained on these generated datasets gradually increases with the dataset progression. A VideoMAE model pre-trained on our synthetic videos closes 97.2\% of the performance gap on UCF101 action classification between training from scratch and self-supervised pre-training from natural videos, and outperforms the pre-trained model on HMDB51. Introducing crops of static images to the pre-training stage results in similar performance to UCF101 pre-training and outperforms the UCF101 pre-trained model on 11 out of 14 out-of-distribution datasets of UCF101-P. Analyzing the low-level properties of the datasets, we identify correlations between frame diversity, frame similarity to natural data, and downstream performance. Our approach provides a more controllable and transparent alternative to video data curation processes for pre-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。