用分形生成大量合成视频,提升动作识别预训练效果。
Pre-training for Action Recognition with Automatically Generated Fractal Datasets
- 基于分形几何自动生成大规模短视频数据集用于预训练。
- 在HMDB51和UCF101上表现接近甚至超越Kinetics预训练。
- 适合缺乏真实标注数据的动作识别研究者使用。
近年来,合成数据在计算机视觉预训练中受到关注,尤其在图像分类、医学影像等任务中展现出潜力。已有研究表明,通过生成过程自动创建的合成样本可替代真实数据并生成强视觉表征,解决数据采集与标注成本高、版权与隐私问题。本文将此趋势拓展至视频领域,应用于动作识别任务。利用分形几何,提出方法自动生成大规模短时合成视频片段,用于神经网络预训练。生成视频具备高度多样性,源于分形固有的多尺度结构生成能力。为缩小域差距,识别真实视频的关键属性,并在预训练中加以模拟。通过详尽消融实验,确定提升下游性能的关键特征,并提供通用预训练指导。在多个基准测试(包括HMDB51、UCF101及四组群组动作、细粒度动作、动态场景任务)上进行微调评估,结果表明该方法性能接近甚至部分优于标准Kinetics预训练。代码与合成视频样本见https://github.com/davidsvy/fractal_video。
原文摘要 · Abstract (English)
In recent years, interest in synthetic data has grown, particularly in the context of pre-training the image modality to support a range of computer vision tasks, including object classification, medical imaging etc. Previous work has demonstrated that synthetic samples, automatically produced by various generative processes, can replace real counterparts and yield strong visual representations. This approach resolves issues associated with real data such as collection and labeling costs, copyright and privacy. We extend this trend to the video domain applying it to the task of action recognition. Employing fractal geometry, we present methods to automatically produce large-scale datasets of short synthetic video clips, which can be utilized for pre-training neural models. The generated video clips are characterized by notable variety, stemmed by the innate ability of fractals to generate complex multi-scale structures. To narrow the domain gap, we further identify key properties of real videos and carefully emulate them during pre-training. Through thorough ablations, we determine the attributes that strengthen downstream results and offer general guidelines for pre-training with synthetic videos. The proposed approach is evaluated by fine-tuning pre-trained models on established action recognition datasets HMDB51 and UCF101 as well as four other video benchmarks related to group action recognition, fine-grained action recognition and dynamic scenes. Compared to standard Kinetics pre-training, our reported results come close and are even superior on a portion of downstream datasets. Code and samples of synthetic videos are available at https://github.com/davidsvy/fractal_video .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。