arXiv:2411.17459cs.CVcs.AI2024-11CVPR被引 65

用小波分解提升视频压缩效率,解决长视频生成的算力瓶颈。

WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent Video Diffusion Model

  • 通过多级小波变换分离视频频率成分,优化低频能量传递。
  • 相比顶尖方法,吞吐量提升2倍,内存降低4倍,重建质量相当。
  • 适合需要高效长视频生成的扩散模型研究者使用。

视频变分自编码器(Video VAE)将视频压缩至低维隐空间,是大多数隐空间视频扩散模型(LVDM)的核心组件,可降低训练成本。然而,随着生成视频分辨率和时长增加,视频VAE的编码开销成为训练LVDM的主要瓶颈。此外,多数LVDM采用分块推理方式,在处理长视频时易导致隐空间不连续。突破计算瓶颈的关键在于将视频分解为不同成分并高效编码关键信息。小波变换可将视频分解为多个频域成分,显著提升编码效率。为此,本文提出小波流自编码器(WF-VAE),利用多级小波变换促进低频能量流入隐表示。同时引入因果缓存(Causal Cache)机制,保障分块推理中隐空间的完整性。相较于现有最优视频VAE,WF-VAE在PSNR和LPIPS指标上表现更优,实现2倍吞吐量提升与4倍内存消耗降低,同时保持竞争力的重建质量。代码与模型已开源:https://github.com/PKU-YuanGroup/WF-VAE。

原文摘要 · Abstract (English)

Video Variational Autoencoder (VAE) encodes videos into a low-dimensional latent space, becoming a key component of most Latent Video Diffusion Models (LVDMs) to reduce model training costs. However, as the resolution and duration of generated videos increase, the encoding cost of Video VAEs becomes a limiting bottleneck in training LVDMs. Moreover, the block-wise inference method adopted by most LVDMs can lead to discontinuities of latent space when processing long-duration videos. The key to addressing the computational bottleneck lies in decomposing videos into distinct components and efficiently encoding the critical information. Wavelet transform can decompose videos into multiple frequency-domain components and improve the efficiency significantly, we thus propose Wavelet Flow VAE (WF-VAE), an autoencoder that leverages multi-level wavelet transform to facilitate low-frequency energy flow into latent representation. Furthermore, we introduce a method called Causal Cache, which maintains the integrity of latent space during block-wise inference. Compared to state-of-the-art video VAEs, WF-VAE demonstrates superior performance in both PSNR and LPIPS metrics, achieving 2x higher throughput and 4x lower memory consumption while maintaining competitive reconstruction quality. Our code and models are available at https://github.com/PKU-YuanGroup/WF-VAE.

视频生成小波变换扩散模型压缩效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。