用扩散模型替代GAN,实现更稳定高效的视频压缩与重建。
Rethinking Video Tokenization: A Conditioned Diffusion-based Approach
- 用条件因果扩散模型代替GAN做视频解码,训练更稳定。
- 仅用MSE+KL+LPIPS损失,单步采样即达顶尖重建效果。
- 支持任意长度视频生成,适合需要高效生成的场景。
现有视频分词器多采用变分自编码器(VAE)架构进行视频压缩与重建,但其训练常依赖复杂多阶段技巧,尤其是最终阶段需精细调节附加生成对抗网络(GAN)的对抗训练,易导致收敛不稳定。相比之下,扩散模型具备更稳定的训练过程和更高质量的生成能力。为此,本文提出一种新型条件扩散式视频分词器CDT,将GAN驱动的解码器替换为条件因果扩散模型。编码器将时空信息压缩为紧凑潜在表示,解码器通过反向扩散过程在潜空间条件生成视频。推理时引入特征缓存机制,实现任意长度视频生成并保持时间连续性,结合采样加速技术提升效率。整个模型仅使用基础MSE扩散损失、KL项及LPIPS感知损失从头训练。大量实验表明,CDT在视频重建任务中达到当前最优性能,且仅需单步采样;其缩放版(推理速度提升3倍)仍可媲美顶级基线。此外,基于CDT训练的潜空间视频生成模型也表现优异。代码与预训练权重已公开于https://github.com/ali-vilab/CDT。
原文摘要 · Abstract (English)
Existing video tokenizers typically use the traditional Variational Autoencoder (VAE) architecture for video compression and reconstruction. However, to achieve good performance, its training process often relies on complex multi-stage training tricks that go beyond basic reconstruction loss and KL regularization. Among these tricks, the most challenging is the precise tuning of adversarial training with additional Generative Adversarial Networks (GANs) in the final stage, which can hinder stable convergence. In contrast to GANs, diffusion models offer more stable training processes and can generate higher-quality results. Inspired by these advantages, we propose CDT, a novel Conditioned Diffusion-based video Tokenizer, that replaces the GAN-based decoder with a conditional causal diffusion model. The encoder compresses spatio-temporal information into compact latents, while the decoder reconstructs videos through a reverse diffusion process conditioned on these latents. During inference, we incorporate a feature cache mechanism to generate videos of arbitrary length while maintaining temporal continuity and adopt sampling acceleration technique to enhance efficiency. Trained using only a basic MSE diffusion loss for reconstruction, along with KL term and LPIPS perceptual loss from scratch, extensive experiments demonstrate that CDT achieves state-of-the-art performance in video reconstruction tasks with just a single-step sampling. Even a scaled-down version of CDT (3$\times$ inference speedup) still performs comparably with top baselines. Moreover, the latent video generation model trained with CDT also exhibits superior performance. The source code and pretrained weights are available at https://github.com/ali-vilab/CDT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。