动态调整视频生成的隐空间帧率,提升效率且几乎不损失质量。
VGDFR: Diffusion-based Video Generation with Dynamic Latent Frame Rate
- 根据运动频率动态调整隐空间帧数,高动量段保留更多细节。
- 实验显示生成速度最高提升3倍,画质损失极小。
- 适合追求高效视频生成的开发者与研究者使用。
基于DiT的视频生成模型虽取得显著成果,但计算开销大。本文发现真实视频具有时序非均匀性,高运动区域信息密度更高。受此启发,提出无需训练的VGDFR方法,通过动态调整隐空间帧率,在低频段减少令牌数量,高频段保留细节。主要贡献包括:(1) 针对DiT视频生成的动态帧率调度器;(2) 一种新的隐空间帧合并方法,确保低分辨率下冗余帧对齐并去噪后合并;(3) 对不同层中旋转位置编码(RoPE)的偏好分析,优化了语义与局部信息捕捉策略。实验表明,VGDFR可实现高达3倍的生成加速,且画质下降可忽略。
原文摘要 · Abstract (English)
Diffusion Transformer(DiT)-based generation models have achieved remarkable success in video generation. However, their inherent computational demands pose significant efficiency challenges. In this paper, we exploit the inherent temporal non-uniformity of real-world videos and observe that videos exhibit dynamic information density, with high-motion segments demanding greater detail preservation than static scenes. Inspired by this temporal non-uniformity, we propose VGDFR, a training-free approach for Diffusion-based Video Generation with Dynamic Latent Frame Rate. VGDFR adaptively adjusts the number of elements in latent space based on the motion frequency of the latent space content, using fewer tokens for low-frequency segments while preserving detail in high-frequency segments. Specifically, our key contributions are: (1) A dynamic frame rate scheduler for DiT video generation that adaptively assigns frame rates for video segments. (2) A novel latent-space frame merging method to align latent representations with their denoised counterparts before merging those redundant in low-resolution space. (3) A preference analysis of Rotary Positional Embeddings (RoPE) across DiT layers, informing a tailored RoPE strategy optimized for semantic and local information capture. Experiments show that VGDFR can achieve a speedup up to 3x for video generation with minimal quality degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。