通过渐进式训练提升视频压缩率,实现更紧凑的潜在空间。
Progressive Growing of Video Tokenizers for Temporally Compact Latent Spaces
- 分阶段训练高压缩模块,利用低压缩模型预训练成果
- 在4倍以上压缩比下仍保持高质量重建效果
- 适合需要高效视频生成的扩散模型研究者
视频标记器对潜在视频扩散模型至关重要,可将原始视频数据转换为时空压缩的潜在空间以实现高效训练。然而,将最先进的视频标记器扩展至超过4倍的时序压缩比而不增加通道容量面临巨大挑战。本文发现,从低压缩编码器恢复的时序下采样视频,其重建质量优于直接对原始视频使用高压缩编码器的结果。这表明高压缩模型可利用低压缩模型的表征。基于此洞察,我们提出一种自举式高压缩时序模型,逐步在已充分训练的低压缩模型基础上训练高压缩模块。方法包含跨层级特征混合模块,保留预训练低压缩模型信息,并引导高压缩模块捕捉完整视频序列的剩余细节。视频基准评估显示,相比直接训练全模型,本方法显著提升了重建质量并增强了时序压缩能力。此外,所得紧凑潜在空间能有效训练视频扩散模型,实现高质量视频生成且大幅降低标记预算。
原文摘要 · Abstract (English)
Video tokenizers are essential for latent video diffusion models, converting raw video data into spatiotemporally compressed latent spaces for efficient training. However, extending state-of-the-art video tokenizers to achieve a temporal compression ratio beyond 4x without increasing channel capacity poses significant challenges. In this work, we propose an alternative approach to enhance temporal compression. We find that the reconstruction quality of temporally subsampled videos from a low-compression encoder surpasses that of high-compression encoders applied to original videos. This indicates that high-compression models can leverage representations from lower-compression models. Building on this insight, we develop a bootstrapped high-temporal-compression model that progressively trains high-compression blocks atop well-trained lower-compression models. Our method includes a cross-level feature-mixing module to retain information from the pretrained low-compression model and guide higher-compression blocks to capture the remaining details from the full video sequence. Evaluation of video benchmarks shows that our method significantly improves reconstruction quality while increasing temporal compression compared to directly training the full model. Furthermore, the resulting compact latent space effectively trains a video diffusion model for high-quality video generation with a significantly reduced token budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。