提出可自适应压缩的1D视频扩散自编码器,提升生成效率与质量。
Adaptive 1D Video Diffusion Autoencoder
- 用可变长度丢弃机制实现动态压缩,节省冗余计算
- 在相同压缩比下重建性能媲美3D-CNN自编码器
- 支持下游生成任务,降低生成伪影,适合视频生成场景
近期视频生成模型依赖将像素空间视频压缩为潜在表示的视频自编码器。然而现有方法存在三大局限:(1) 固定率压缩,对简单视频浪费令牌;(2) 架构僵化,难以建模变长潜在表示;(3) 确定性解码器难以从压缩潜在中恢复细节。为此,我们提出一维扩散视频自编码器(One-DVA),一种基于Transformer的自适应1D编码与扩散解码框架。编码器采用基于查询的视觉变压器提取时空特征并生成潜在表示,结合可变长度丢弃机制动态调整潜在长度。解码器为像素空间扩散变压器,以潜在表示作为输入条件重建视频。通过两阶段训练策略,One-DVA在相同压缩比下达到与3D-CNN VAE相当的重建性能。更重要的是,其支持自适应压缩,实现更高压缩率。为进一步支持下游潜在生成,我们对One-DVA潜在分布进行正则化,并微调解码器以缓解生成过程中的伪影。
原文摘要 · Abstract (English)
Recent video generation models largely rely on video autoencoders that compress pixel-space videos into latent representations. However, existing video autoencoders suffer from three major limitations: (1) fixed-rate compression that wastes tokens on simple videos, (2) inflexible CNN architectures that prevent variable-length latent modeling, and (3) deterministic decoders that struggle to recover appropriate details from compressed latents. To address these issues, we propose One-Dimensional Diffusion Video Autoencoder (One-DVA), a transformer-based framework for adaptive 1D encoding and diffusion-based decoding. The encoder employs query-based vision transformers to extract spatiotemporal features and produce latent representations, while a variable-length dropout mechanism dynamically adjusts the latent length. The decoder is a pixel-space diffusion transformer that reconstructs videos with the latents as input conditions. With a two-stage training strategy, One-DVA achieves performance comparable to 3D-CNN VAEs on reconstruction metrics at identical compression ratios. More importantly, it supports adaptive compression and thus can achieve higher compression ratios. To better support downstream latent generation, we further regularize the One-DVA latent distribution for generative modeling and fine-tune its decoder to mitigate artifacts caused by the generation process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。