用轻量化VAE解码器加速视频生成,速度提升6倍且质量几乎不变。
Flash-VAED: Plug-and-Play VAE Decoders for Efficient Video Generation
- 通过通道剪枝和操作优化,减少解码器冗余计算。
- 实现约6倍加速,重建质量保持在96.9%以上。
- 适合需要高效视频生成的开发者与研究者使用。
潜在扩散模型已实现高质量视频合成,但其推理过程仍耗时昂贵。随着扩散变压器日益高效,延迟瓶颈逐渐转移到VAE解码器。为在保持质量的同时降低其延迟,我们提出一个通用的VAE解码器加速框架,保留与原始潜在分布的完全对齐。具体包括:(1) 一种考虑通道独立性的剪枝方法,有效缓解严重通道冗余;(2) 针对广泛使用的因果3D卷积高推理开销的分阶段主导算子优化策略。基于此构建了Flash-VAED系列。此外,设计了三阶段动态蒸馏框架,高效将原VAE解码器能力迁移至Flash-VAED。在Wan和LTX-Video VAE解码器上的大量实验表明,本方法在质量和速度上均优于基线,实现约6×加速,同时重建性能维持在96.9%。显著地,Flash-VAED使端到端生成流程提速最高达36%,在VBench-2.0上质量损失可忽略。代码已开源。
原文摘要 · Abstract (English)
Latent diffusion models have enabled high-quality video synthesis, yet their inference remains costly and time-consuming. As diffusion transformers become increasingly efficient, the latency bottleneck inevitably shifts to VAE decoders. To reduce their latency while maintaining quality, we propose a universal acceleration framework for VAE decoders that preserves full alignment with the original latent distribution. Specifically, we propose (1) an independence-aware channel pruning method to effectively mitigate severe channel redundancy, and (2) a stage-wise dominant operator optimization strategy to address the high inference cost of the widely used causal 3D convolutions in VAE decoders. Based on these innovations, we construct a Flash-VAED family. Moreover, we design a three-phase dynamic distillation framework that efficiently transfers the capabilities of the original VAE decoder to Flash-VAED. Extensive experiments on Wan and LTX-Video VAE decoders demonstrate that our method outperforms baselines in both quality and speed, achieving approximately a 6$\times$ speedup while maintaining the reconstruction performance up to 96.9%. Notably, Flash-VAED accelerates the end-to-end generation pipeline by up to 36% with negligible quality drops on VBench-2.0. Our code is available at https://github.com/Aoko955/Flash-VAED.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。