用生成式解码器实现32倍视频压缩,兼顾生成质量
REGEN: Learning Compact Video Embedding with (Re-)Generative Decoder
- 用扩散Transformer生成缺失细节,替代传统精确重建
- 实现最高32倍时间压缩率,比顶尖方法高8倍
- 适合需要高效文本到视频生成的场景
我们提出一种新的视频嵌入学习视角:视频嵌入器无需精确复现输入视频,只需合成视觉上合理的重构结果即可。这一宽松准则使压缩比大幅提升,且不影响下游生成模型质量。具体而言,我们用编码器-生成器框架替代传统编码器-解码器结构,采用扩散Transformer(DiT)从紧凑潜在空间中合成缺失细节,并设计专用潜在条件模块,使DiT解码器基于编码后的视频潜在嵌入进行生成。实验表明,该方法在压缩比提升时仍优于现有最先进方法。为验证有效性,我们报告了视频嵌入器在高达32倍的时间压缩比下的表现(比领先方法高8倍),并验证了该超紧凑潜在空间在文本到视频生成中的鲁棒性,显著提升了潜在扩散模型的训练与推理效率。
原文摘要 · Abstract (English)
We present a novel perspective on learning video embedders for generative modeling: rather than requiring an exact reproduction of an input video, an effective embedder should focus on synthesizing visually plausible reconstructions. This relaxed criterion enables substantial improvements in compression ratios without compromising the quality of downstream generative models. Specifically, we propose replacing the conventional encoder-decoder video embedder with an encoder-generator framework that employs a diffusion transformer (DiT) to synthesize missing details from a compact latent space. Therein, we develop a dedicated latent conditioning module to condition the DiT decoder on the encoded video latent embedding. Our experiments demonstrate that our approach enables superior encoding-decoding performance compared to state-of-the-art methods, particularly as the compression ratio increases. To demonstrate the efficacy of our approach, we report results from our video embedders achieving a temporal compression ratio of up to 32x (8x higher than leading video embedders) and validate the robustness of this ultra-compact latent space for text-to-video generation, providing a significant efficiency boost in latent diffusion model training and inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。