通过跳过重复帧的计算，让视频生成更快更省力。
Latent Inter-Frame Pruning: A Training-Free Method Bridging Traditional Video Compression and Modern Diffusion Transformers for Efficient Generation
- 在潜在空间中跳过冗余帧的重计算，无需训练
- 提速1.44倍，实测达12.44帧/秒
- 适合追求高效生成的开发者和研究者
视频生成虽能生成逼真视频，但计算成本高、速度慢，难以支持实时应用。本文观察到，在潜空间扩散模型（LDM）框架下，视频潜在表示沿时间轴存在冗余。类比传统视频压缩避免传输重复帧数据，我们提出潜空间帧间剪枝（Latent Inter-Frame Pruning）框架，跳过重复潜变量块的重计算，从而降低计算负担并提升吞吐量。然而直接剪枝会导致视觉伪影，因训练与推理序列不一致。为此，我们设计注意力恢复机制以弥合训练-推理差距。实验表明，该方法使视频编辑吞吐量提升1.44倍，在NVIDIA RTX 6000上实现12.44 FPS，同时保持视频质量。本工作为将传统视频压缩技术与现代生成流水线融合提供了初步思路。
原文摘要 · Abstract (English)
Video generation, while capable of generating realistic videos, is computationally expensive and slow, prohibiting real-time applications. In this paper, we observe that video latents encoded via an autoencoder under the Latent Diffusion Model (LDM) framework contain redundancy along the temporal axis. Analogous to how traditional video compression algorithms avoid transmitting redundant frame data, we propose the Latent Inter-frame Pruning framework to prune (skip the re-computation of) duplicated latent patches, thereby reducing computational burden and increasing throughput. However, direct pruning results in visual artifacts due to the discrepancy between full-sequence training and pruned inference. To resolve these artifacts, we propose an Attention Recovery mechanism to bridge the train-inference gap. With our proposed method, we increase video editing throughput by 1.44$\times$, achieving 12.44 FPS on an NVIDIA RTX 6000 while maintaining video quality. We hope our work inspires further research into integrating traditional video compression methods with modern video generation pipelines. This work is a preliminary work on Training-free Latent Inter-Frame Pruning with Attention Recovery.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。