用关键帧压缩+重建,仅用万分之一数据达到全量训练效果
VideoCompressa: Data-Efficient Video Understanding via Joint Temporal Compression and Spatial Reconstruction
- 通过可微关键帧选择与变分自编码器压缩视频帧
- 在UCF101上仅用0.13%数据超全量训练2.34个百分点
- 适合资源受限下视频模型训练,尤其对大模型微调有帮助
视频理解模型的可扩展性正受大规模视频数据集存储与计算成本的制约。尽管数据合成已在图像领域提升数据效率,但其向视频领域的延伸仍因普遍存在的时序冗余和复杂的时空动态而困难重重。本文揭示一个关键洞察:视频数据集低效的主要来源并非样本间冗余,而是样本内帧级冗余。为此,提出VideoCompressa框架,将问题重构为动态潜在压缩。该框架联合优化一个轻量级卷积网络实现的可微关键帧选择器(采用Gumbel-Softmax采样),以及一个预训练冻结的变分自编码器(VAE),将选中的关键帧压缩为紧凑且语义丰富的潜在码。这些潜在表示输入压缩网络,支持端到端反向传播。关键帧选择器与合成潜在码协同优化,以最大化保留任务相关信息。实验表明,该方法实现前所未有的数据效率:在UCF101上使用仅0.13%原始数据,基于ConvNets的模型超越全量训练2.34个百分点,较传统合成方法提速超过5800倍;在HMDB51上微调Qwen2.5-7B-VL,仅需0.41%训练数据即达全量训练性能,优于零样本基线10.61个百分点。
原文摘要 · Abstract (English)
The scalability of video understanding models is increasingly limited by the prohibitive storage and computational costs of large-scale video datasets. While data synthesis has improved data efficiency in the image domain, its extension to video remains challenging due to pervasive temporal redundancy and complex spatiotemporal dynamics. In this work, we uncover a critical insight: the primary source of inefficiency in video datasets is not inter-sample redundancy, but intra-sample frame-level redundancy. To leverage this insight, we introduce VideoCompressa, a novel framework for video data synthesis that reframes the problem as dynamic latent compression. Specifically, VideoCompressa jointly optimizes a differentiable keyframe selector-implemented as a lightweight ConvNet with Gumbel-Softmax sampling-to identify the most informative frames, and a pretrained, frozen Variational Autoencoder (VAE) to compress these frames into compact, semantically rich latent codes. These latent representations are then fed into a compression network, enabling end-to-end backpropagation. Crucially, the keyframe selector and synthetic latent codes are co-optimized to maximize retention of task-relevant information. Experiments show that our method achieves unprecedented data efficiency: on UCF101 with ConvNets, VideoCompressa surpasses full-data training by 2.34\% points using only 0.13\% of the original data, with over 5800x speedup compared to traditional synthesis method. Moreover, when fine-tuning Qwen2.5-7B-VL on HMDB51, VideoCompressa matches full-data performance using just 0.41\% of the training data-outperforming zero-shot baseline by 10.61\%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。