arXiv:2409.06665cs.CV2024-09ECCV被引 1

用静态图像生成伪视频,实现无需真实数据的视频预训练。

Data Collection-free Masked Video Modeling

论文配图:Data Collection-free Masked Video Modeling
图 1 · 摘自论文原文
  • 从静态图像生成伪运动视频,替代真实视频数据
  • 在动作识别任务上超越仅用静态图像的方法,接近真实+合成数据效果
  • 适合缺乏视频数据资源或关注隐私合规的研究者

视频变换器的预训练通常需要大量数据,带来高昂的数据收集成本以及隐私、版权和固有偏见等问题。数据合成是解决这些问题的可行路径,但仅依赖合成数据仍存在挑战。本文提出一种有效的自监督学习框架,利用现成且低成本的静态图像进行视频预训练。具体地,设计了伪运动生成器(PMG)模块,通过递归应用图像变换生成伪运动视频,并用于掩码视频建模。该方法同样适用于合成图像,从而完全摆脱视频数据收集的成本与风险。在动作识别任务上的实验表明,该框架能有效学习时空特征,显著优于仅使用静态图像的方法,部分情况下甚至超过同时使用真实与合成视频的方法。这些结果揭示了视频变换器在掩码视频建模中所学内容的片段。

原文摘要 · Abstract (English)

Pre-training video transformers generally requires a large amount of data, presenting significant challenges in terms of data collection costs and concerns related to privacy, licensing, and inherent biases. Synthesizing data is one of the promising ways to solve these issues, yet pre-training solely on synthetic data has its own challenges. In this paper, we introduce an effective self-supervised learning framework for videos that leverages readily available and less costly static images. Specifically, we define the Pseudo Motion Generator (PMG) module that recursively applies image transformations to generate pseudo-motion videos from images. These pseudo-motion videos are then leveraged in masked video modeling. Our approach is applicable to synthetic images as well, thus entirely freeing video pre-training from data collection costs and other concerns in real data. Through experiments in action recognition tasks, we demonstrate that this framework allows effective learning of spatio-temporal features through pseudo-motion videos, significantly improving over existing methods which also use static images and partially outperforming those using both real and synthetic videos. These results uncover fragments of what video transformers learn through masked video modeling.

自监督学习视频建模数据合成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。