通过保留运动动态的深层结构,实现视频生成模型高效压缩。
Individual Content and Motion Dynamics Preserved Pruning for Video Diffusion Models
- 按内容与运动特性分层剪枝,浅层删冗余块,深层保关键结构。
- 在多个基准上保持视频质量的同时,推理速度提升1.25至2.5倍。
- 适合需快速生成高质量视频的实时应用或资源受限场景。
视频扩散模型(VDMs)因计算开销大、推理慢而难以部署。本文提出一种新压缩方法:基于个体内容与运动动态保留的剪枝策略及一致性损失。实验发现,深层网络对运动连贯性至关重要,浅层更关注单帧内容。因此,我们剪除浅层冗余模块,保留深层结构,得到轻量版模型VDMini。同时引入个体内容与运动动态(ICMD)一致性损失:先用个体内容蒸馏损失保持每帧特征一致;再用多帧内容对抗损失增强整体视频运动流畅性。在文本到视频(T2V)和图像到视频(I2V)任务中,分别实现对SF-V、T2V-Turbo-v2、HunyuanVideo的平均1.25×、1.4×、2.5×加速,且在UCF101、VBench-T2V、VBench-I2V等基准上保持生成质量。
原文摘要 · Abstract (English)
The high computational cost and slow inference time are major obstacles to deploying Video Diffusion Models (VDMs). To overcome this, we introduce a new Video Diffusion Model Compression approach using individual content and motion dynamics preserved pruning and consistency loss. First, we empirically observe that deeper VDM layers are crucial for maintaining the quality of \textbf{motion dynamics} (\textit{e.g.,} coherence of the entire video), while shallower layers are more focused on \textbf{individual content} (\textit{e.g.,} individual frames). Therefore, we prune redundant blocks from the shallower layers while preserving more of the deeper layers, resulting in a lightweight VDM variant called VDMini. Moreover, we propose an \textbf{Individual Content and Motion Dynamics (ICMD)} Consistency Loss to gain comparable generation performance as larger VDM to VDMini. In particular, we first use the Individual Content Distillation (ICD) Loss to preserve the consistency in the features of each generated frame between the teacher and student models. Next, we introduce a Multi-frame Content Adversarial (MCA) Loss to enhance the motion dynamics across the generated video as a whole. This method significantly accelerates inference time while maintaining high-quality video generation. Extensive experiments demonstrate the effectiveness of our VDMini on two important video generation tasks, Text-to-Video (T2V) and Image-to-Video (I2V), where we respectively achieve an average 2.5 $\times$, 1.4 $\times$, and 1.25 $\times$ speed up for the I2V method SF-V, the T2V method T2V-Turbo-v2, and the T2V method HunyuanVideo, while maintaining the quality of the generated videos on several benchmarks including UCF101, VBench-T2V, and VBench-I2V.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。