轻量级视频自编码器，让视频生成更快更省算力
LeanVAE: An Ultra-Efficient Reconstruction VAE for Video Diffusion Models
- 用邻域感知模块和不重叠块操作降低计算开销
- 结合小波变换与压缩感知，重建质量更高
- 推理速度比现有方法快44倍，算力需求降50倍
近期基于潜在空间的视频扩散模型（LVDMs）通过视频变分自编码器（Video VAEs）将复杂视频数据压缩至紧凑潜在空间，推动了视频生成技术的发展。然而，随着LVDM训练规模扩大，视频自编码器的计算开销成为关键瓶颈，尤其在高分辨率视频编码时更为显著。为此，我们提出LeanVAE，一种新型超高效视频自编码器框架，包含两项核心创新：(1) 基于邻域感知前馈（NAF）模块和非重叠块操作的轻量级架构，大幅降低计算成本；(2) 融合小波变换与压缩感知技术，提升重建质量。大量实验证明，LeanVAE在视频重建与生成方面均优于现有视频自编码器，尤其在效率上表现突出：相比现有方法，最高可减少50倍浮点运算量（FLOPs），推理速度提升44倍，同时保持具有竞争力的重建质量，为可扩展、高效的视频生成提供了新思路。模型与代码已开源：https://github.com/westlake-repl/LeanVAE
原文摘要 · Abstract (English)
Recent advances in Latent Video Diffusion Models (LVDMs) have revolutionized video generation by leveraging Video Variational Autoencoders (Video VAEs) to compress intricate video data into a compact latent space. However, as LVDM training scales, the computational overhead of Video VAEs becomes a critical bottleneck, particularly for encoding high-resolution videos. To address this, we propose LeanVAE, a novel and ultra-efficient Video VAE framework that introduces two key innovations: (1) a lightweight architecture based on a Neighborhood-Aware Feedforward (NAF) module and non-overlapping patch operations, drastically reducing computational cost, and (2) the integration of wavelet transforms and compressed sensing techniques to enhance reconstruction quality. Extensive experiments validate LeanVAE's superiority in video reconstruction and generation, particularly in enhancing efficiency over existing Video VAEs. Our model offers up to 50x fewer FLOPs and 44x faster inference speed while maintaining competitive reconstruction quality, providing insights for scalable, efficient video generation. Our models and code are available at https://github.com/westlake-repl/LeanVAE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。