通过优化视频VAE的潜在空间结构，显著提升扩散模型训练速度与生成质量。
Delving into Latent Spectral Biasing of Video VAEs for Superior Diffusability
- 设计两种轻量正则化方法，引导潜在空间呈现低频主导的时空谱特性。
- 在文本到视频生成中实现3倍训练加速，视频奖励提升10%。
- 适用于希望加速扩散模型训练的研究者与开发者。
潜变量扩散模型将VAE与扩散主干结合，而VAE潜在空间的结构直接影响扩散训练难度。现有视频VAE多关注重建保真度，忽视潜在空间结构。本文对视频VAE潜在空间进行统计分析，识别出两个关键谱特性：时空中频谱偏向低频，通道内特征谱由少数主导模式构成。为此，提出两种轻量、与主干无关的正则化方法：局部相关性正则化与潜在掩码重建。实验表明，所提出的谱结构化VAE（SSVAE）在文本到视频生成任务中实现3倍收敛速度提升，视频奖励提高10%，优于多个开源强基线模型。代码已公开于https://github.com/zai-org/SSVAE。
原文摘要 · Abstract (English)
Latent diffusion models pair VAEs with diffusion backbones, and the structure of VAE latents strongly influences the difficulty of diffusion training. However, existing video VAEs typically focus on reconstruction fidelity, overlooking latent structure. We present a statistical analysis of video VAE latent spaces and identify two spectral properties essential for diffusion training: a spatio-temporal frequency spectrum biased toward low frequencies, and a channel-wise eigenspectrum dominated by a few modes. To induce these properties, we propose two lightweight, backbone-agnostic regularizers: Local Correlation Regularization and Latent Masked Reconstruction. Experiments show that our Spectral-Structured VAE (SSVAE) achieves a $3\times$ speedup in text-to-video generation convergence and a 10\% gain in video reward, outperforming strong open-source VAEs. The code is available at https://github.com/zai-org/SSVAE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。